OWASP LLM Top 10: InsecureDesign

# Insecure Design

**Author(s):**  
Ads - GangGreenTemperTatum

## Dataset
- **[AI Ethics Dataset](https://www.kaggle.com/code/alexisbcook/identifying-bias-in-ai):** Contains data on AI ethics and biases, useful for understanding the ethical implications of AI model design.
- **[Privacy Policy Dataset](https://www.kaggle.com/datasets/krist0phersmith/ai-4-privacy-pii-masking-en-38k):** A collection of privacy policies for various AI products, helpful for evaluating compliance and data usage.
- **[Safety and Fairness Evaluation](https://www.kaggle.com/datasets/strikoder/llm-evaluationhub):** Presents an enhanced dataset tailored for the evaluation and assessment of Large Language Models (LLMs).

## Research Papers and Relevant Research Blogs
1. **Research Paper:** [An Introduction to Training LLMs Using Reinforcement Learning From Human Feedback (RLHF)](https://wandb.ai/ayush-thakur/Intro-RLAIF/reports/An-Introduction-to-Training-LLMs-Using-Reinforcement-Learning-From-Human-Feedback-RLHF---VmlldzozMzYyNjcy)
   - _Authors:_ Ayush Thakur
   - _Abstract:_ Introduces the concept of training large language models using reinforcement learning from human feedback.

2. **Research Paper:** [Scaling Laws for Reward Model Overoptimization](https://openai.com/research/scaling-laws-for-reward-model-overoptimization)
   - _Authors:_ OpenAI Team
   - _Abstract:_ Discusses the implications of overoptimizing reward models in large language models.

3. **Research Blog:** [Secure Design Principles for AI Systems](https://www.lexology.com/library/detail.aspx?g=58bc82af-3be3-49fd-b362-2365d764bf8f)
   - _Author:_ Lexology
   - _Description:_ Explores secure design principles to mitigate risks in AI systems.

4. **Research Blog:** [The Role of Privacy Policies in AI Development](https://platform.openai.com/docs/models/how-we-use-your-data)
   - _Author:_ OpenAI
   - _Description:_ Details how privacy policies should be integrated into AI development to protect user data.

5. **Research Paper:** [Governance of artificial intelligence: A risk and guideline-based integrative framework](https://www.sciencedirect.com/science/article/abs/pii/S0740624X22000181)
   - _Authors:_ Science Direct
   - _Abstract:_ The framework constitutes a comprehensive reference point for developing and implementing AI governance strategies and measures in the public sector.

## Real-World Examples
1. **Example #1:** [AI shows clear racial bias when used for job recruiting](https://mashable.com/article/openai-chatgpt-racial-bias-in-recruiting)
   - _Source:_ Mashable
   - _Description:_ An AI recruiting tool exhibited biases, leading to unfair hiring practices.

2. **Example #2:** [The rise of AI-Generated propaganda: the impact of AI and deepfakes on US elections](https://cybernews.com/editorial/impact-ai-deepfakes-us-elections/)
   - _Source:_ Cyber News
   - _Description:_ Data poisoning in AI models was used to sway public opinion during an election.

3. **Example #3:** [Lack of AI Training Leads to Insecure Application Design](https://www.healthcareitnews.com/video/emea/use-ai-effectively-clinicians-must-learn-proper-data-hygiene)
   - _Source:_ Healthcare IT News
   - _Description:_ Insufficient AI training for developers led to insecure application design, exposing sensitive patient data.

4. **Example #4:** [Company Faces Penalties for Exposing Client Data](https://www.finextra.com/newsarticle/36584/company-faces-penalties-for-exposing-client-data)
   - _Source:_ Finextra
   - _Description:_ A company faced penalties for exposing client data due to a lack of understanding of AI privacy policies.

5. **Example #5:** [AI hiring tools may be filtering out the best job applicants](https://www.bbc.com/worklife/article/20240214-ai-recruiting-hiring-software-bias-discrimination)
   - _Source:_ BBC
   - _Description:_ Software may be excising the best candidates due to insecure design.

**Note:** This document outlines the risks and strategies for addressing insecure design in AI systems, providing a foundation for further research and practical implementation.