I need to go do this compeition the instructions re: # Challenge Overview

---

Our nation suddenly faces an unprecedented threat from the invading forces of Red Land. To counter this, the Singapore Armed Forces (SAF) has initiated **"Operation Guardian,"** which ****deploys a cutting-edge defence system: **Guardian Turret** across our nation for a robust civilian defence. These turrets, designed for civilian use, are to be mounted atop buildings all across the country, forming a formidable network of defence.

### The Challenge Statement

However, operating a military-grade turret isn't everyone's forte. Therefore, SAF has innovated a solution: **voice-controlled operation**. This is where yeou, a talented AI engineer, come into play. Embark on a learning journey in AI and autonomous robotics and compete with other teams by deploying **computer vision and natural language processing models** to complete challenges in real-world scenarios. Through this, you will be deploying **autonomous air defence turrets** that can **identify, classify and take down air threats** to defend our skies.

### Your Mission

You are tasked with developing an **AI algorithm that can interpret short, spoken commands and translate them into precise actions for the turre**t. This includes identifying targets and selecting appropriate ammunition. The challenge is to make this system intuitive and efficient, so even those with no military training can effectively defend our land.

# Challenge Tasks

---

### Automatic Speech Recognition (ASR)

Given an audio file of a turret command instruction, return a transcription of the instruction.

Note that noise simulating the corruption of radio transmissions will be present in the audio datasets provided to **both Novice and Advanced teams.** 

### Natural Language Processing (NLP)

Given an audio transcription of a turret command instruction, return a JSON object corresponding to the specified target’s description, heading, and the tool to be deployed against it.

**For Advanced teams**, the transcript will be a turret instruction in natural language. 

**For Novice teams**, the transcript will follow a relatively structured format with the turret operator explicitly reading out the turret, heading, and tool, though the order may vary.

### Vision-Language Model (VLM) Task

Given an image and a caption describing a target in that image, return a bounding box corresponding to the target’s location within the image. 

Note that targets within a given image are not uniquely identified by their object class (e.g. ”airplane”, “helicopter”); multiple targets within an image may be members of the same object class. Instead, targets provided will correspond to a particular target description (e.g. “black and white drone”). 

Not all possible target descriptions will be represented in the training dataset provided to participants. There will also be unseen targets and novel descriptions in the test data used in the hidden test cases of the Virtual Qualifiers, Semi-Finals / Finals. As such, Guardians will have to develop vision models capable of understanding **natural language** to identify the correct target from the scene.

# Dataset Details

---

While “Operation Guardian” aims to implement a state-of-the-art defence system using autonomous air defence turrets, the Guardians start to face significant challenges : 

- Communication between the Command and Control Centre (C2) and the turrets is faltering due to **increasing corruption in radio transmissions**. Control room commands are met with a barrage of **static radio communications noise**, which intensifies each day - rendering the instructions barely inaudible
- The turret-mounted robots, which have been in service for some time, are showing signs of sensor degradation. This results in progressively **noisier digital images**, complicating the visual data the Guardians rely on. While robots for Novice Guardians have been replaced, **those for Advanced Guardians have not, forcing them to contend with worsening visual clarity.**

### Dataset Augmentation

In real-life scenarios, it is common for the distribution of incoming deployment data to deviate from the distribution of the data the model was trained on. As a result of data drift, your training data will not perfectly match the data presented to you in subsequent rounds. In light of this, **you will have to ensure your submitted models are robust enough to still make good predictions on noisy out-of-distribution data.** 

While your team is free to use whatever libraries you wish for data augmentation, we suggest the `[audiomentations](https://iver56.github.io/audiomentations/)` library for audio augmentation and the `[albumentations](https://albumentations.ai/docs/)` library for image augmentation due to their ease of use.

### Miscellaneous Details

- Audio files are provided in .WAV format with a sample rate of 16 kHz. Images are provided as 1520x870 JPG files.
- In the **audio datasets** provided to both the Novice and Advanced Guardians, noise will be present. Guardians who wish to fine-tune their models on additional data are free to use the (clean, unaugmented) National Speech Corpus data present in the `til-ai-24-data` bucket on Google Cloud Storage.
- As for the **image datasets** provided to both Novice and Advanced Guardians, there will be no noise present. However, it is worth noting that your models will have to be adequately robust as the hidden test cases for the Virtual Qualifiers and the Semi-Finals/Finals will have increasing amounts of noise introduced. This is especially crucial for **Advanced Guardians**, due to the degradation of their robot sensors.

# Submission Process

---

### For Qualifiers, all submissions will be handled through the [**GCP Online Development Environment](https://console.cloud.google.com/vertex-ai/workbench/instances?project=dsta-angelhack) .**

- Additional instructions on how to work within the development environment are available in the provided [**Educational Resources](https://www.notion.so/Resources-and-FAQ-2b8c22465bc343f2ba698d14c635e6f7?pvs=21)**
- For each of the qualifier competition tasks (i.e. ASR, NLP, and VLM), teams will be expected to use CLI commands to submit a Docker image with your submitted model packaged inside, for a total of 3 distinct Docker images to be submitted. Detailed step-by-step instructions are included in the [competition base code repository](https://github.com/TIL-24/til-24-base). These submitted Docker images will be automatically run and evaluated against our hidden test cases, with your scores automatically being updated to our live competition leaderboard **[coming soon]**
    - We will have 500 hidden test cases for each of ASR and NLP, and 1000 hidden test cases for VLM.
- Submission Process for Semi-Finals & Finals are **classified** (for now) 
****

<aside>
<img src="https://cdn-icons-png.flaticon.com/512/7154/7154370.png" alt="https://cdn-icons-png.flaticon.com/512/7154/7154370.png" width="40px" /> **REMINDER:** All teams are to submit before **26 May 2024, 11:59:59 PM SGT/GMT +8** for the Virtual Qualifiers. If you face any issues or require help during submission, contact us on the `#helpdesk` channel on the **TIL-AI Discord Server**, or email us at <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>.

</aside>

# Prizes

---

### **Novice Track:**

- **1st Place:** $5,000
- **2nd Place:** $3,000
- **3rd Place:** $1,500
- **Developer's Choice Award:** $250

### **Advanced Track:**

- **1st Place** : $10,000
- **2nd Place:** $7,500
- **3rd Place:** $4,250
- **Developer's Choice Award:** $500

### **What is a Developer Choice Award ? 🏅**

A special recognition given to **1 team (per track)** at the Finals. This award recognises the team that most effectively encapsulates the spirit of innovation, strategic execution, and collaborative effort in their project through a succinct **3-minute pitch** done before their Finals showcase. 

The winners will be determined only by the **votes of onsite attendees at the Finals*** 

**Finalist teams are eligible to vote but with a **cross-track voting rule** (i.e. ****Advanced teams can only vote for Novice teams and vice versa).*

- **How should teams craft their pitch?**
    1. **Innovative Approach and Strategy**: How well the team articulates the unique methods and optimisations they implemented to enhance the performance of their AI model. This includes the rationale behind their choices and the impact on the model’s speed and accuracy.
    2. **Resilience in Facing Challenges**: The ability of the team to identify significant obstacles and their ingenuity in adapting their strategies to overcome these challenges. This criterion highlights the team's resilience and adaptability.
    3. **Reflective Insights and Achievements**: The depth of insights and learnings shared about their competition experience, including notable achievements. This reflects how the project has contributed to their growth and expertise in AI development. 

# Judging Criteria

---

## Qualifiers

For the automated scoring of your submitted models, 25% of your team’s score will be based on the performance of each of your Automatic Speech Recognition (ASR) and Natural Language Processing (NLP) models, with the remaining 50% based on the performance of your team’s Vision-Language Model (VLM). These percentages break down further, with 90% coming from the accuracy of each submitted model’s predictions, and the final 10% based on its speed independent of accuracy.

A detailed breakdown of the scoring criteria is as follows:

- ASR (25%)
    - Accuracy (1 – WER) — 22.5%
    - Speed ($t_{\max}$ 1.5hrs)— 2.5%
- NLP (25%)
    - Accuracy (Exact Match for heading, ROUGE-L for other fields) — 22.5%
    - Speed ($t_{\max}$ 1.5hrs)— 2.5%
- VLM (50%)
    - Accuracy (IoU @ 0.5) — 45%
        - If the output bounding box from the model has IoU ≥ 0.5 against our ground truth, it will be scored as correct. The overall accuracy will thus be calculated as $\frac{\text{num correct}}{\text{num in total}}$
    - Speed ($t_{\max}$ 1.5hrs)— 5%

For further details on the accuracy scoring calculations, you can see the functions that will be used to evaluate your model’s predictions in the [competition base code repository](https://github.com/TIL-24/til-24-base). For further details on the speed scoring calculations, see the [Speed](https://www.notion.so/Project-Submission-74caa4421c5e42bc9bc044111b63025d?pvs=21) section below.

## Semi-Finals & Finals

Similar to previous rounds, 10% of your team’s score will be based on the speed at which your combined system makes predictions, with the remaining 90% of your team’s score determined by the accuracy of the predictions made. This 90% is split evenly between the combined ASR/NLP component as well as the VLM component. Note that while autonomy performance is not scored directly, poor autonomy/robotics code will likely lead to poor performance across the board.

A detailed breakdown of the scoring criteria for the Semi-Finals is as follows:

- ASR + NLP Accuracy (Exact Match for heading, ROUGE-L for other fields) — 45%
    - Note that while ASR accuracy is not scored, without a good ASR prediction, NLP accuracy will likely suffer!
- VLM Accuracy (IoU @ 0.5) — 45%
    - Note that if the target heading/autonomy code is wrong, the target object may not be visible or only partially visible in the resulting simulator snapshot. The target bounding box will be scored based on its total size, whether the whole target object is visible in the image taken or not.
- Overall Speed ($t_{\max}$ TBA)— 10%

The top 14 teams (7 from each track) based on their scores as calculated above will automatically proceed to the Finals.

## Speed

Scores for model speed are calculated by taking the time taken by your model $t_{\text{taken}}$ and the defined maximum $t_{\max}$ (e.g. 1.5hrs for Qualifiers), and feeding them through this formula.

$$
1-\frac{\min(t_{\text{taken}},t_{\max})}{t_{\max}}
$$

In short, if your model takes longer than $t_{\max}$ to complete, your score for speed will be 0. The less time your model takes to generate its predictions, the better your resulting score, with linearly better scores based on your model’s inference speed, with a perfect score requiring instantaneous results. Note that for the Virtual Qualifiers, $t_{\text{taken}}$ includes container startup time (anywhere from 10-20 mins depending on your container’s size and how long it takes to initialize).