# I Can't Believe It's Not Better: Challenges in Applied Deep Learning

Why don’t deep learning approaches always deliver as expected in the real world?

Dive deep into the pitfalls and challenges of applied deep learning.

In recent years, we have witnessed a remarkable rise of deep learning (DL), whose impressive performance on benchmark tasks has led to increasing ambitions to deploy DL in real-world applications across all fields and disciplines [1, 2, 3, 4, 5]. However, despite its potential, DL still faces many challenges during deployment in dynamic, real-world conditions, exposing practical limitations that are often overlooked in controlled benchmarks.


Current publication mechanisms tend to prioritize solutions that work on standard bench, lacking a platform to systematically collect real-world failure cases. Moreover, discussions about these failures are usually confined within specific domains, with limited cross-domain interaction, even though these failures may have similar underlying causes. Establishing a platform for collecting and sharing real-world challenges and failures of DL can address fundamental issues to facilitate more successful deployment of DL across domains, and enhance understanding of theoretical and empirical weaknesses in machine learning (ML) research.


Building such a platform and fostering this community has been the continuous goal of our I Can’t Believe It’s Not Better (ICBINB) initiative. As DL systems have become increasingly present in everyday life also for non-scientific people, we want to put a special focus on real-world applications now. Therefore, in this proposed ICBINB workshop, we aim to explore the challenges, unexpected outcomes, and common principles underlying similar issues and failure modes encountered across various fields and disciplines when deploying DL models in real-world scenarios. We will focus the discussion on:


Challenges & failure modes: We will invite papers from diverse fields including but not limited to healthcare, scientific discovery, robotics, education, equality & fairness, and social sciences to discuss the challenges and failure modes when deploying DL models for domain-specific applications as well as the underlying reasons. The failure modes may include suboptimal performance, concerns with the safety and reliability of applying DL models in unpredictable real-world applications, as well as ethical and societal challenges.


Common challenges across domains & underlying reasons: We aim to discuss common reasons or patterns in challenges and failure modes across disciplines, which may include, but are not limited to, data-related issues (e.g., distribution shift, bias, label quality), model limitations (e.g., ethics, fairness, interpretability, scalability, domain alignment), and deployment challenges (e.g., computational demands, hardware constraints).


This workshop forms one workshop in a series as part of the larger I Can't Believe It's Not Better (ICBINB) activities. We are a diverse group of researchers promoting the idea that there is more to machine learning research than tables with bold numbers. We believe that understanding in machine learning can come through more routes than iteratively improving upon previous methods and as such this workshop aims to focus on understanding through negative results. Previous workshops have focused on ideas motivated by beauty and gaps between theory and practice in probabilistic ML, we also run a monthly seminar series aiming to crack open the research process and showcase what goes on behind the curtain. Read more about our activities and our members here.


We invite researchers and industry professionals to submit their papers on negative results, failed experiments, and unexpected challenges encountered in applying deep learning to real-world problems across industry and science. The primary goal of this workshop is to create a platform for open and honest discussion about the hurdles and roadblocks in applying deep learning. We believe that sharing these experiences is crucial for the advancement of the field, providing valuable insights that can prevent others from repeating the same mistakes and fostering a culture of transparency and learning. We invite submissions from novel, ongoing, and unpublished research that apply deep learning to various domains including, but not limited to, social sciences, biology, physics, chemistry, engineering, robotics, psychology, healthcare, neuroscience, marketing, economics, or finance. Submitted papers should contain the following four elements:

- A use case that was tackled with deep learning. 

- A solution for this type of use case was proposed in the deep learning literature

- A description of the (negative) outcome in the solution. 

- An investigation (and ideally an answer) to the question of why it did not work as promised by the deep learning literature. 

The potential reasons for failure may include but are not limited to data-related issues (e.g., distribution shift, bias, label quality, noisy measurement, quality of simulated data), model limitations (e.g., assumption violations, robustness, interpretability, scalability, representation misalignment), and deployment challenges (e.g., computational demands, hardware constraints). Besides these four points, papers will be assessed on:

- Rigor and transparency in the scientific methodologies employed. 

- Novelty and significance of insights.

- Quality of discussion of limitations.

- Reproducibility of results.

- Clarity of writing.