# Results from the Turing Seminar hackathon

We ([EffiSciences](https://ia.effisciences.org/)) ran a hackathon at the end of [the Turing Seminar in ENS Paris-Saclay](https://www.master-mva.com/cours/seminaire-turing/) and [ENS Ulm](https://diplome.di.ens.fr/catalog_fr.html#:~:text=niveau%20L3%20M1-,S%C3%A9minaire%20Turing,-(iCal)), an academic course inspired by the AGISF, with 28 projects submitted by 44 participants between the 11th and 12th November.

We share a selection of projects.  **See them all **[**here**](https://drive.google.com/drive/folders/1HgqUCu7CG44iUZdJtDO_Zua4xjLFu450?usp=drive_link)**.**

I think some of them could even be turned into valuable blog posts, and I’ve learnt a lot by reading everything. Here are a few extracts.

Towards Monosemanticity: Decomposing Vision Models with Dictionary Learning
===========================================================================

*David HEURTEL-DEPEIGES* \[[link](https://drive.google.com/file/d/1gmj-ax9mtn2chjyoMxyQlP-R9zCl6q6f/view?usp=sharing)\]

Basically an adaptation of the famous dictionary learning paper on vision CNN.

“When looking at 100 random **features**, 46 were found to be interpretable and monosemantic, \[...\] When doing the same experiment with 100 random **neurons**, 2 were found to be interpretable and monosemantic”. Very good execution.

Open, closed, and everything in between: towards human-centric regulations for safer large models
=================================================================================================

*Théo SAULUS* \[[link](https://drive.google.com/file/d/1jopH_aaDOVRSLoBG4qnwbqglDUyFebUH/view)\]

Imho, an almost SOTA summary of the current discourse on open sourcing models.

A Review of The Debate on Open-Sourcing Powerful AI Models
==========================================================

*Tu Duyen NGUYEN, Adrien RAMANANA RAHARY* \[[link](https://drive.google.com/file/d/1M1B7n2jEPZPkBdYSxy_v3uNgqWAk9tF4/view?usp=drive_link)\]

“What is the goal of this document? Our goal is to clarify, and sometimes criticize, the arguments regularly put forward in this, both from the proponents and the opponents of open-sourced AI models. In an effort to better understand both stances, we have classified the most common arguments surrounding this debate in five main topics:

1.  Safety evaluation of powerful AI models through audits or open-research
2.  Competition in the private sector and beyond
3.  Safety risks of open-sourcing strongly capable models
4.  Preserving open science for its own sake
5.  The feasibility of closing the weights of AI models

In an effort to highlight the strengths and weaknesses of each stance, we will present and challenge for each family the arguments of both sides, as well as include in each family some arguments which we think are relevant, but have not been mentioned in most discussions on these topics.”

They tried to be exhaustive, but there are still some gaps. The format is nevertheless interesting (even if it could be even more concise).

| ![](https://lh7-us.googleusercontent.com/jPqOl0fdtsn9gJqt2T-z4pj3ruxzjOUsRTc8AfNZ9II17pNFC6EPxDObUBhH4n_nSD0bEJnO5ZPoIZgt2OOZbpIbY3oDxUdvVbDC6nkpoLGrptMDNHPFeQzyl64AATU3ng7WTg0xH_tWpkJTcEDz_TU) | ![](https://lh7-us.googleusercontent.com/V8fz6uv9YM2-ZXvTv2HSosgPJasXQsHVsB5JLIPAtsb7tW3XTkX8z430kpt1F6_K27eI-LzagFxX3Yxd-uW-WVR92BK2y8H17E8Kze2WVhYbLVf-yUNmMH2QBSBoB_1vzLos9gLB6Fv9-88HfMH1yzM) | ![](https://lh7-us.googleusercontent.com/ve6FN23edQDd1xv_5RbzWl6CpFFlf1Yo6A9xcmnlqOKrGo3j2pgBZGwYiRVmOhkKxbwgFFDlf6hpfmWvTIIlfjVe-SATK8S35b_qKhtUV01y5Yvt72gsdkK5r059M579OTe0hxPRoWcjBfASmi9Ldcw) | ![](https://lh7-us.googleusercontent.com/lJo2tdjyZbu-C3rzMCQe5osMFmJVoiH5qJ3mPWCu7rrHIxt64NcRYXJ9V7v5faVLj1aGodrZ6UerX6OHdgpTNbqjSa83L0MJ9dfaDZkMDqd-T5hiYIfJns50S4zdo1wVa5muFO2r2uiKVhnYUFDl4ig) |
| --- | --- | --- | --- |

AI safety media analysis
========================

*Gurvan Richardeau, Raphaël Pesah* \[[link](https://docs.google.com/document/d/1SBdZN9vnSVfqqJ_zUChXPxQjcrv8bT5DqSubA20ob-w/edit#heading=h.u543ptgy0th)\]

“The first analysis we’ve done involved examining major themes by employing text data analysis and ML methods (such as tf-idf analysis and topic modeling) within a corpus of 1,644 publications from the Factiva database over the past five years, specifically related to AI safety. The second analysis (Analysis 2) uses another database: Europresse. “

Here are some snippets:

“First let’s have a look at the global distribution of the articles over the years.”

| We see that more than 80% of the articles of our dataset are from 2023: ![](https://lh7-us.googleusercontent.com/xxQRvJ3_VrXmCXxlLoGIQ3w17nrgpUWVBvMURWoOrBiSTYRTkYJ_RttRslCnwTsFX4mNifVYACCKeX7ACdhPOKeDV1tUrD1KzggYqJgj5h1EYO0PIAFIveXlcQTWQdzj4xQ5PQrfoFOtiNQHjt6SN6M) | Now let’s have a more precise look over the year 2023 : ![](https://lh7-us.googleusercontent.com/Snm5vx1Vw0kusjyo0ARoB5KPRP7TDjwDdEkUMKUYnRblkHnnaEaE6fDHKalZq_G9OIcbgd1wjDNm1bnF25PsuethBK1lgHxDuZBPQaFXB4zCKfVy9bRuY17f1u300DWf3kvg5FnuX7jnwMlS1nCDyUU) |
| --- | --- |

  
“Here are the results for the sentiment analysis: “

![](https://39669.cdn.cke-cs.com/rQvD3VnunXZu34m86e5f/images/df4676c4d743b93c1e54c3d61373b011be1f56fe50ef5ac5.png)

“Around 71% of the articles talking of AGI talk about it as a good thing whereas 23% are talking about it as a thing to worry about.”

“We got 57,000 articles about AGI in 2023 against 18,000 for the ones related to AI safety, so this time it is three times less. \[...\] There are between two and three times more articles talking about AGI without AI safety than articles about AI safety in 2023.”

My comment: Interesting. There are many more figures in the reports. Maybe those kinds of metrics could be used to measure the impact of public outreach?

Biases in Reinforcement Learning from Human Feedback
====================================================

*Gaspard Berthelier* \[[link](https://docs.google.com/document/d/1Qc7f21EgYHdFJqC5MGJu3Auj00vLfJEc/edit?usp=drive_link)\]

A good summary of the paper “[Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://www.lesswrong.com/posts/LqRD7sNcpkA9cmXLv/open-problems-and-fundamental-limitations-of-rlhf)”.

Situational Awareness of AIs
============================

*Thomas Michel, Théo Rudkiewicz* \[[link](https://docs.google.com/document/d/1v-L1nXVSEJw0_9ceAA5BQGJxRgoSk9dvjbEat8eSQQE/edit?usp=drive_link)\]

An alternative title could be “Situational Awareness from AI to Zombies, with premium pedagogical memes”:

| ![](https://lh7-us.googleusercontent.com/iRRbeMtQDyt6xvZZrEUGzcwQL0KtdpjkoqFapPbKaUMZayyF9NnhmQd5JrUGyoIz37z4wbfLRzgmkYw_CJjAJwKQCqcDr0wfI7p39j_otKGvoGq6FucBkiVdcmlU4v-ki5VkImxC8-j0htjy8ojzZ1Y) Fig. 1: Why an IA that does not differentiate testing and deployment is less dangerous. | ![](https://lh7-us.googleusercontent.com/pQFiVS6maF_dj4ENrw57i0CAd5zpFuYp1GcDOua-CWloFrW5xZ0Wt1U0u2k6JmL5kXhsyQCRKavK9KJVchUB_rNPSvFpFfkBatm1Da0JYS6oLJRlrqqmIecx9f03qFp5VpwTFvvcBHQdONOh8ntOXjA) Fig. 2: What is Situational Awareness? |
| --- | --- |
| ![](https://lh7-us.googleusercontent.com/Um-l5OrkcBfvdTI4jnEn4F3vISh7yeOfQ12iJMYaLHAIVJ386OL1q7PudydH8VmZroBdsASvBSW_Co-CvAH9pGajcdSTSFlxIXUZAe4jwmbqSPfEKdSq444TkvLJlEX99Mv8PR9fV14UwO6SQsLq25k) Fig. 3: How to test potentially dishonest models? | ![](https://lh7-us.googleusercontent.com/CwXqj6bcsUaVH5VlX83vXSEOvJ4QaDzRDOChNVUBBM1_Yf0UMhbVARhgc-lelR02BWxx_UIw8x0_Q5Ypw-vPvvrxq7ZINMbwaQrnpdWMNlah6fSrtM-Wtg5MbfnQ0ra86nU9jOrKrco7qTWtkSkUNJo) Fig. 4: A summary of [Berglund et al](https://arxiv.org/abs/2309.00667): Taken out of context: On measuring situational awareness in LLMs. |

Summary of methods to solve Goal Misgeneralization
==================================================

*Vincent Bardusco* \[[link](https://docs.google.com/document/d/16SlCd52xOuCYLOCtASobc_EXuauVOWwt/edit?usp=drive_link)\]

A good summary of the following papers:

1.  R. Shah, Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals, 2022.
2.  B. Shlegeris, The prototypical catastrophic AI action is getting root access to its datacenter, 2022.
3.  Song et al., Constructing unrestricted adversarial examples with generative models, 2018.
4.  A. Bhattad, M. J. Chong, K. Liang, B. Li, D. A and Forsyth, Unrestricted Adversarial Examples via Semantic Manipulation, 2019.
5.  B. Barnes, Imitative Generalisation (AKA 'Learning the Prior'), 2021.
6.  R. Jia and P. Liang, Adversarial examples for evaluating reading comprehension systems, 2017.
7.  S. Goldwasser, M. P. Kim, V. Vaikuntanathan and O. Zamir, Planting undetectable backdoors in machine learning models, 2022.
8.  A. Madry, A. Makelov, L. Schmidt, D. Tsipras and A. Vladu, Towards deep learning models resistant to adversarial attacks, 2018.
9.  E. Hubinger, Relaxed adversarial training for inner alignment, 2019. 

A Review of DeepMind’s AI alignment plan
========================================

*Antoine Poirier, Théo Communal* \[[link](https://drive.google.com/file/d/1dC6FreML2qs1S6xuopIfYzVATRYc5lC8/view?usp=drive_link)\]

A document summarizing the main aspects of DeepMind's plan. Up until now, their agenda has been covered in a series of blog posts and papers, but now it's all summarized in a ten-page blog post.

| ![](https://lh7-us.googleusercontent.com/v5VJbubRuSrQrOtjV5ULHV9NuooiQ6MhZV3hcfeUv-gPMRI9aRmek2BjM2eyZ3hZE4wdvAvb2DJllf29nLk7cJIIKL--IBh28jIXzC5VpLZcIHyqjT2hY8WCPCcAUlrmsS5zw81DZXLGhFAx4jQ3aSs) |   |
| --- | --- |
| ![](https://lh7-us.googleusercontent.com/c4yUlNNmtZbcqSx88_mLX3MW1GtNUmk9qv48b_a51-QAE6lCglsOtz9lEJui7L8CTUh9WkSBtkj4zmvDTgYdCB0_4cBZ_NKFj2OlgGcnMcrlwRCy0wXhZsIA_Qradf3EqZF8bOS-6wC1kPJSGz8xJww) | ![](https://lh7-us.googleusercontent.com/-2nEmUmgll5XqSt9fkiBI0ZtvlxkHDXVGXwuf8n6T1i3daRTDWVncFcmbRu_Z1xihh0X6cX6j5_YuCmGiUTBh91jVaqI0F8WLuv5IsxSsy1I1VvjndlLMeSpOo1JESmBog5hVoik7Rq23Cr_ahbAdGg) |

Criticism of criticism of interp
================================

*Gabriel Ben Zenou, Joachim Collin* \[[link](https://docs.google.com/document/d/1EFTdstgHb8iVIUx_bWzmcYNDZUCzbP3gpbMQmfZLc6M/edit?usp=drive_link)\]

An alternative title could be “Against [Against Almost Every Theory of Impact of Interpretability](https://www.lesswrong.com/posts/LNA8mubrByG7SFacm/against-almost-every-theory-of-impact-of-interpretability-1)”.

Some students tried to distill the discussion and to criticize my position, and you can find my criticism of their criticism in the comments of the google doc. Here is my [main comment](https://docs.google.com/document/d/1EFTdstgHb8iVIUx_bWzmcYNDZUCzbP3gpbMQmfZLc6M/edit?disco=AAABBpJ9O7A).

Risks of Value Lock-In in China
===============================

*Inès Larroche, Bastien Le Chenadec* \[[link](https://drive.google.com/file/d/1a81ezHWDE10gnABsk8V_7iEreSx0024S/view?usp=drive_link)\]

It’s a very good summary of what is happening in China regarding AI. 

Is LeCun making progress in AI safety with “A Path Towards Autonomous Machine Intelligence”?
============================================================================================

*Victor Morand* \[[link](https://drive.google.com/file/d/1muZiF9R6sxPU2Z-gMsMF0j60kMT8CT5P/view?usp=drive_link)\]

It’s a good summary of LeCun’s idea, and the numerous criticisms of his plan.

![](https://lh7-us.googleusercontent.com/BpaVRJvOA_ZspcWoearixkKK0Bd9HqnRUWV2UMiweIY5K-8AOAPdPDYIf1LUuDSHrPOxuJuiPfyFMNcXAfm_nEBHVPDfXic-5n5zXr3sZKBA0DxeTluI155BbcQHDPbHsqkK7W4OlYZRIr2-hBdOC7o)

LeCun's architecture

Taxonomy of governance regulations
==================================

Mathis Embit \[[link](https://docs.google.com/document/d/1VtLmxM9mdBTIvKY0rWNIGtjB8ODdVwgYZ0OuX4BGyXs/edit?usp=drive_link)\]

A summary of the main ways to regulate AI. Comparing different policies provides them with more distinctiveness and depth.

![](https://lh7-us.googleusercontent.com/XkN12GBH2rKj5I4UPtQhHH2-7t12kbr7d3j8UKSjWM0yDhPQhsm7a43unvGbFuvf4O1CgCyGd6fp9oqVQLWyhxcQAXwJ0RWs4jUmrAwBK-DTSAtcCsJP12rjN-nNcbPqlKJB_TixonVzG4bxEZSQYq8)

Other notable projects
======================

 See them all [here](https://drive.google.com/drive/folders/1HgqUCu7CG44iUZdJtDO_Zua4xjLFu450?usp=drive_link).

| **Title** | **Authors** | **Summary** |
| --- | --- | --- |
| Technical justification of OpenAI's superalignment plan | Robin Sobczyk, Bastien Lhopitallier | A lot of papers are summarized here |
| Agent Creation | Mathias Vigouroux | A philosophical essay trying to explain how theory of mind emerges in humans and could emerge in AIs. This is very interesting. |
| Anthropic RSP analysis | Ralf Cortes Do Nascimento, Paul Cazali | A good summary of the entire LW discussion on the RSPs. |
| Goal Misgeneralization | Basile TERVER, Antoine OLIVIER | A summary of various techniques to prevent Goal Misgeneralization |
| Detecting and reducing gender bias | Gabrielle LE BELLIER | A literature review on techniques to reduce gender biases in LLMs |
| Literature review on SOTA Adversarial attacks and backdoors | Mathis Le Bail, David Sahna | A summary of many papers on attack and defense. The conclusion: "all the studies in the literature tend to the following conclusion: there is no single effective method of defense against all possible types of attack" |
| Why does DL work so well? | Sacha ELKOUBI, Even MATENCIO, Eustache LE BIHAN, Paul SITOLEUX | It needs a bit more polishing, but this is a good basis. It goes from explaining the Classical Statistical learning theory to Singular learning theory and everything in between. |
| AI Act: what rules for foundation models? | Matthieu Carreau | A summary of the current challenges for EU regulations |
| Critics of GovAI | Xavier Secheresse, Pierre-Yves Doireau | A summary of their views, and a small critique |
| Cure | David El Bèze, Raphaël Cohen, Nathaniel Cohen | A short story of a deceptive AI transforming the world |
| Logit Lens on Vision Transformer | Akedjou Achraff Adjileye | Testing the logit lens on Vision transformers, it works without difficulty. |
| Safety benchmarks | Ugo Insalaco | A review of the main safety benchmarks and a few proposals of missing benchmarks. |
| Probability of X-Risks without deceptive alignment | Mathilde DUPOUY | She enumerates the different scenarios in the following [table](https://docs.google.com/spreadsheets/d/1W2sm4Si9zu6i9ioBbV_vnC6RTCenrPXRJwitvn9gqYg/edit?usp=sharing). The final probability among all the risks that the one that occurs is an existential risk without deceptive alignment is 71%. |
| Do you think Le Cun's architecture is safe? | Théotime de Charrin | Another criticism of LeCun's agenda. |
| Navigating AI Safety | Abdessalam EDDIB | Presents a landscape of some strategies to reduce risks through alignment techniques and some governance strategies, and is targeting beginners in alignment. |

Some Thoughts
=============

*   Running the hackathon was a lot of fun. We just worked in a room at the university during the weekend. We also organized a meme contest. Highly recommended.
*   **Unlike last year's hackathon, where students had the freedom to choose their own topics, this time we provided a list of predetermined topics**. This resulted in higher quality work. If you're interested in organizing a similar hackathon, you can find the list of subjects[ here](https://www.notion.so/AI-Safety-Research-Projets-2023-English-b6c5b1cb3ad641abbfa9acd856cd8bad?pvs=21) if you want to run something similar.
*   I am extremely pleased with the outcome, especially since the students went from having no knowledge in AI Safety to making valuable contributions in only 2 months.
*   The material of the course is available [here](https://docs.google.com/document/d/1d0W_5xRaVehYO_j4P1TN4-JJGktMNEXgBxO4MXCr68o/edit#heading=h.s5znjvp3l4zi).