hey so 
Problem Statement
Feature Extraction from Images
In this hackathon, the goal is to create a machine learning model that extracts entity values from images. This capability is crucial in fields like healthcare, e-commerce, and content moderation, where precise product information is vital. As digital marketplaces expand, many products lack detailed textual descriptions, making it essential to obtain key details directly from images. These images provide important information such as weight, volume, voltage, wattage, dimensions, and many more, which are critical for digital stores.
Data Description:
The dataset consists of the following columns:
index: A unique identifier (ID) for the data sample.
image_link: Public URL where the product image is available for download. Example link - https://m.media-amazon.com/images/I/71XfHPR36-L.jpg  To download images, use the download_images function from src/utils.py. See sample code in src/test.ipynb.
group_id: Category code of the product.
entity_name: Product entity name. For example, “item_weight”.
entity_value: Product entity value. For example, “34 gram”.
Note: For test.csv, you will not see the column entity_value as it is the target variable.
Output Format:
The output file should be a CSV with 2 columns:
index: The unique identifier (ID) of the data sample. Note that the index should match the test record index.
prediction: A string which should have the following format: “x unit” where x is a float number in standard formatting and unit is one of the allowed units (allowed units are mentioned in the Appendix). The two values should be concatenated and have a space between them.
For example: “2 gram”, “12.5 centimetre”, “2.56 ounce” are valid.
Invalid cases: “2 gms”, “60 ounce/1.7 kilogram”, “2.2e2 kilogram”, etc.
Note: Make sure to output a prediction for all indices. If no value is found in the image for any test sample, return an empty string, i.e., “”. If you have less/more number of output samples in the output file as compared to test.csv, your output won’t be evaluated.
File Descriptions:
Source Files:
src/sanity.py: Sanity checker to ensure that the final output file passes all formatting checks.
Note: The script will not check if fewer/more number of predictions are present compared to the test file. See sample code in src/test.ipynb.
src/utils.py: Contains helper functions for downloading images from the image_link.
src/constants.py: Contains the allowed units for each entity type.
sample_code.py: A sample dummy code that can generate an output file in the given format. Usage of this file is optional.
Dataset Files:
dataset/train.csv: Training file with labels (entity_value).
dataset/test.csv: Test file without output labels (entity_value). Generate predictions using your model/solution on this file's data and format the output file to match sample_test_out.csv (Refer to the "Output Format" section above).
dataset/sample_test.csv: Sample test input file.
dataset/sample_test_out.csv: Sample outputs for sample_test.csv. The output for test.csv must be formatted in the exact same way.
Note: The predictions in the file might not be correct.
Constraints:
You will be provided with a sample output file and a sanity checker file. Format your output to match the sample output file exactly and pass it through the sanity checker to ensure its validity.
Note: If the file does not pass through the sanity checker, it will not be evaluated. You should receive a message like Parsing successful for file: ...csv if the output file is correctly formatted.
You are given the list of allowed units in constants.py and also in the Appendix. Your outputs must be in these units. Predictions using any other units will be considered invalid during validation.

this is constants.py
entity_unit_map = {
    'width': {'centimetre', 'foot', 'inch', 'metre', 'millimetre', 'yard'},
    'depth': {'centimetre', 'foot', 'inch', 'metre', 'millimetre', 'yard'},
    'height': {'centimetre', 'foot', 'inch', 'metre', 'millimetre', 'yard'},
    'item_weight': {'gram',
        'kilogram',
        'microgram',
        'milligram',
        'ounce',
        'pound',
        'ton'},
    'maximum_weight_recommendation': {'gram',
        'kilogram',
        'microgram',
        'milligram',
        'ounce',
        'pound',
        'ton'},
    'voltage': {'kilovolt', 'millivolt', 'volt'},
    'wattage': {'kilowatt', 'watt'},
    'item_volume': {'centilitre',
        'cubic foot',
        'cubic inch',
        'cup',
        'decilitre',
        'fluid ounce',
        'gallon',
        'imperial gallon',
        'litre',
        'microlitre',
        'millilitre',
        'pint',
        'quart'}
}

allowed_units = {unit for entity in entity_unit_map for unit in entity_unit_map[entity]}


this is utils.py
import re
import constants
import os
import requests
import pandas as pd
import multiprocessing
import time
from time import time as timer
from tqdm import tqdm
import numpy as np
from pathlib import Path
from functools import partial
import requests
import urllib
from PIL import Image

def common_mistake(unit):
    if unit in constants.allowed_units:
        return unit
    if unit.replace('ter', 'tre') in constants.allowed_units:
        return unit.replace('ter', 'tre')
    if unit.replace('feet', 'foot') in constants.allowed_units:
        return unit.replace('feet', 'foot')
    return unit

def parse_string(s):
    s_stripped = "" if s==None or str(s)=='nan' else s.strip()
    if s_stripped == "":
        return None, None
    pattern = re.compile(r'^-?\d+(\.\d+)?\s+[a-zA-Z\s]+$')
    if not pattern.match(s_stripped):
        raise ValueError("Invalid format in {}".format(s))
    parts = s_stripped.split(maxsplit=1)
    number = float(parts[0])
    unit = common_mistake(parts[1])
    if unit not in constants.allowed_units:
        raise ValueError("Invalid unit [{}] found in {}. Allowed units: {}".format(
            unit, s, constants.allowed_units))
    return number, unit


def create_placeholder_image(image_save_path):
    try:
        placeholder_image = Image.new('RGB', (100, 100), color='black')
        placeholder_image.save(image_save_path)
    except Exception as e:
        return

def download_image(image_link, save_folder, retries=3, delay=3):
    if not isinstance(image_link, str):
        return

    filename = Path(image_link).name
    image_save_path = os.path.join(save_folder, filename)

    if os.path.exists(image_save_path):
        return

    for _ in range(retries):
        try:
            urllib.request.urlretrieve(image_link, image_save_path)
            return
        except:
            time.sleep(delay)
    
    create_placeholder_image(image_save_path) #Create a black placeholder image for invalid links/images

def download_images(image_links, download_folder, allow_multiprocessing=True):
    if not os.path.exists(download_folder):
        os.makedirs(download_folder)

    if allow_multiprocessing:
        download_image_partial = partial(
            download_image, save_folder=download_folder, retries=3, delay=3)

        with multiprocessing.Pool(64) as pool:
            list(tqdm(pool.imap(download_image_partial, image_links), total=len(image_links)))
            pool.close()
            pool.join()
    else:
        for image_link in tqdm(image_links, total=len(image_links)):
            download_image(image_link, save_folder=download_folder, retries=3, delay=3)

this is my test csv
index,image_link,group_id,entity_name
0,https://m.media-amazon.com/images/I/110EibNyclL.jpg,156839,height
1,https://m.media-amazon.com/images/I/11TU2clswzL.jpg,792578,width
2,https://m.media-amazon.com/images/I/11TU2clswzL.jpg,792578,height

this is how my output should be 
index,prediction
0,21.9 foot
1,10 foot
2,
3,289.52 kilovolt
4,1078.99 kilowatt
        


ok so i have made this script
import re
import cv2
import pytesseract
import pandas as pd
import requests
from io import BytesIO
from PIL import Image
import numpy as np
from concurrent.futures import ProcessPoolExecutor, as_completed
import multiprocessing
from functools import partial
import tqdm
import logging
import os

# ------------------------------
# Configuration and Mappings
# ------------------------------

# Configure logging
logging.basicConfig(
    filename='ocr_pipeline.log',
    filemode='a',
    format='%(asctime)s - %(levelname)s - %(message)s',
    level=logging.INFO
)

# Configure Tesseract executable path
pytesseract.pytesseract.tesseract_cmd = '/opt/homebrew/bin/tesseract'

# Entity to unit mappings
entity_unit_map = {
    'width': {'centimetre', 'cm', 'foot', 'ft', 'inch', 'in', 'metre', 'm', 'millimetre', 'mm', 'yard', 'yd'},
    'depth': {'centimetre', 'cm', 'foot', 'ft', 'inch', 'in', 'metre', 'm', 'millimetre', 'mm', 'yard', 'yd'},
    'height': {'centimetre', 'cm', 'foot', 'ft', 'inch', 'in', 'metre', 'm', 'millimetre', 'mm', 'yard', 'yd'},
    'item_weight': {'gram', 'g', 'kilogram', 'kg', 'microgram', 'µg', 'milligram', 'mg', 'ounce', 'oz', 'pound', 'lb', 'ton', 't'},
    'maximum_weight_recommendation': {'gram', 'g', 'kilogram', 'kg', 'microgram', 'µg', 'milligram', 'mg', 'ounce', 'oz', 'pound', 'lb', 'ton', 't'},
    'voltage': {'kilovolt', 'kV', 'millivolt', 'mV', 'volt', 'V'},
    'wattage': {'kilowatt', 'kW', 'watt', 'W'},
    'item_volume': {
        'centilitre', 'cL', 'cubic foot', 'cubic inch', 'cup', 'decilitre', 'dL', 'fluid ounce', 'fl oz',
        'gallon', 'gal', 'imperial gallon', 'litre', 'L', 'microlitre', 'μL', 'millilitre', 'mL', 
        'pint', 'pt', 'quart', 'qt'
    }
}

# Abbreviation mappings
abbreviation_map = {
    'cm': 'centimetre', 'm': 'metre', 'mm': 'millimetre', 'ft': 'foot', 'in': 'inch', 'yd': 'yard',
    'kg': 'kilogram', 'g': 'gram', 'mg': 'milligram', 'µg': 'microgram', 'oz': 'ounce', 'lb': 'pound', 't': 'ton',
    'kV': 'kilovolt', 'mV': 'millivolt', 'V': 'volt',
    'kW': 'kilowatt', 'W': 'watt',
    'L': 'litre', 'mL': 'millilitre', 'cL': 'centilitre', 'dL': 'decilitre', 'μL': 'microlitre',
    'gal': 'gallon', 'fl oz': 'fluid ounce', 'pt': 'pint', 'qt': 'quart', 'cup': 'cup',
    'cubic foot': 'cubic foot', 'cubic inch': 'cubic inch', 'imperial gallon': 'imperial gallon'
}

# Entity keywords for identification
entity_keywords = {
    'width': ['width', 'breadth', 'wide'],
    'depth': ['depth'],
    'height': ['height', 'length', 'len'],
    'item_weight': ['weight', 'wt'],
    'maximum_weight_recommendation': ['maximum weight', 'recommended weight', 'max weight', 'max wt', 'maximum wt'],
    'voltage': ['voltage'],
    'wattage': ['wattage'],
    'item_volume': ['capacity', 'volume']
}

# ------------------------------
# Helper Functions
# ------------------------------

def preprocess_image(img, max_size=1024):
    """
    Resize image if the largest dimension exceeds max_size.
    """
    height, width = img.shape[:2]
    if max(height, width) > max_size:
        scaling_factor = max_size / float(max(height, width))
        img = cv2.resize(img, None, fx=scaling_factor, fy=scaling_factor, interpolation=cv2.INTER_AREA)
    return img

def extract_numerical_values(text, entity):
    """
    Extract numerical values and their units based on the entity.
    """
    units = entity_unit_map.get(entity, set())
    if not units:
        return 'void'
    
    # Escape units that have special regex characters
    units_escaped = [re.escape(unit) for unit in units]
    pattern = r'(\d+(.\d+)?)\s*(' + '|'.join(units_escaped) + r')\b'
    matches = re.findall(pattern, text, re.IGNORECASE)

    if matches:
        standardized_matches = [
            (
                float(match[0]),
                abbreviation_map.get(match[2].lower(), match[2].lower())
            ) for match in matches
        ]
        return standardized_matches
    return 'void'

def extract_dimensions_pattern(text):
    """
    Extract dimensions if present in the format 'LxWxH unit'.
    """
    pattern = r'(\d+(?:.\d+)?)x(\d+(?:.\d+)?)x(\d+(?:.\d+)?)\s*(mm|cm|in|ft|m)'
    match = re.search(pattern, text, re.IGNORECASE)

    if match:
        length_value = float(match.group(1))
        width_value = float(match.group(2))
        height_value = float(match.group(3))
        unit = abbreviation_map.get(match.group(4).lower(), match.group(4).lower())

        return {
            'length': (length_value, unit),
            'width': (width_value, unit),
            'height': (height_value, unit)
        }
    return {}

def extract_entity_values(text, query_entities):
    """
    Extract values for the specified entities from the given text.
    """
    entity_matches = {entity: 'void' for entity in query_entities}

    # Check for dimension patterns first
    dimension_matches = extract_dimensions_pattern(text)
    if dimension_matches:
        for entity in query_entities:
            if entity in dimension_matches:
                entity_matches[entity] = dimension_matches[entity]

    # Extract based on keywords if not found in dimensions
    for entity, keywords in entity_keywords.items():
        if entity in query_entities and entity_matches[entity] == 'void':
            for keyword in keywords:
                if re.search(r'\b' + re.escape(keyword) + r'\b', text, re.IGNORECASE):
                    numerical_values = extract_numerical_values(text, entity)
                    if numerical_values != 'void':
                        entity_matches[entity] = numerical_values
                        break

    # Fallback to any numerical values if still not found
    for entity in query_entities:
        if entity_matches[entity] == 'void':
            unit_matches = extract_numerical_values(text, entity)
            if unit_matches != 'void':
                entity_matches[entity] = unit_matches

    return entity_matches

def ocr_extract_entities(image_data, query_entities):
    """
    Perform OCR on the image data and extract specified entities.
    """
    try:
        img = Image.open(BytesIO(image_data))
        img = cv2.cvtColor(np.array(img), cv2.COLOR_RGB2BGR)
        img = preprocess_image(img)

        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
        # Apply thresholding to binarize the image
        _, thresh1 = cv2.threshold(gray, 0, 255, cv2.THRESH_OTSU | cv2.THRESH_BINARY_INV)

        # Define a rectangular kernel for dilation
        rect_kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (18, 18))
        dilation = cv2.dilate(thresh1, rect_kernel, iterations=1)

        # Find contours
        contours, _ = cv2.findContours(dilation, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
        extracted_text = ""

        # Custom Tesseract configuration for speed
        custom_config = r'--oem 3 --psm 6 -c tessedit_char_whitelist=0123456789.,cmftinmlyozkgsVW'

        for cnt in contours:
            x, y, w, h = cv2.boundingRect(cnt)
            cropped = img[y:y + h, x:x + w]
            text = pytesseract.image_to_string(cropped, config=custom_config)
            extracted_text += text + "\n"

        extracted_entities = extract_entity_values(extracted_text, query_entities)
        return extracted_entities
    except Exception as e:
        logging.error(f"OCR extraction failed: {e}")
        return {entity: 'void' for entity in query_entities}

def process_row(row):
    """
    Process a single row: download image, perform OCR, extract entity value.
    Returns a dictionary with index and prediction.
    """
    image_link = row['image_link']
    entity_name = row['entity_name']
    query_entities = [entity_name]

    try:
        response = requests.get(image_link, timeout=10)
        if response.status_code == 200:
            image_data = response.content
            extracted_entities = ocr_extract_entities(image_data, query_entities)
            entity_value = extracted_entities.get(entity_name, 'void')

            # Format the prediction based on extracted entity value
            if entity_value != 'void' and entity_value != []:
                if isinstance(entity_value, list):
                    numerical_value = f"{entity_value[0][0]:,.2f}"
                    unit = entity_value[0][1].lower()
                    prediction = f"{numerical_value} {unit}"
                elif isinstance(entity_value, tuple):
                    numerical_value = f"{entity_value[0]:,.2f}"
                    unit = entity_value[1].lower()
                    prediction = f"{numerical_value} {unit}"
                elif isinstance(entity_value, (float, int)):
                    prediction = f"{entity_value:,.2f}"
                else:
                    prediction = ''
            else:
                prediction = ''
        else:
            logging.warning(f"Failed to download image from {image_link}, Status Code: {response.status_code}")
            prediction = ''
    except Exception as e:
        logging.error(f"Error processing image {image_link}: {e}")
        prediction = ''

    return {'index': row.name, 'prediction': prediction}

# ------------------------------
# Main Processing Function
# ------------------------------

def main():
    input_file = '/Users/kartik/Downloads/student_resource 3/dataset/test.csv'
    output_file = 'output_final.csv'

    # Read input CSV without limiting rows
    try:
        df = pd.read_csv(input_file)
        logging.info(f"Successfully read {len(df)} entries from {input_file}")
    except Exception as e:
        logging.critical(f"Failed to read input file {input_file}: {e}")
        return

    # Prepare the output CSV
    # If the output file exists, remove it to avoid appending to old data
    if os.path.exists(output_file):
        os.remove(output_file)
        logging.info(f"Existing output file {output_file} found and removed.")

    # Define batch size for incremental saving
    batch_size = 10000  # Adjust based on available memory and performance
    total_entries = len(df)
    logging.info(f"Starting OCR processing of {total_entries} entries with batch size {batch_size}.")

    # Determine the number of workers based on CPU cores
    num_workers = multiprocessing.cpu_count()
    logging.info(f"Using {num_workers} worker processes for parallel processing.")

    # Open the output CSV and write headers
    with open(output_file, 'w') as f_out:
        f_out.write('index,prediction\n')

    # Create a generator to yield rows
    def row_generator(df):
        for _, row in df.iterrows():
            yield row

    # Process rows in batches
    with ProcessPoolExecutor(max_workers=num_workers) as executor:
        batch = []
        results = []
        futures = {}
        row_gen = row_generator(df)
        pbar = tqdm.tqdm(total=total_entries, desc="Processing Rows", unit="row")

        while True:
            try:
                # Fill the batch
                while len(batch) < batch_size:
                    row = next(row_gen)
                    batch.append(row)
            except StopIteration:
                # End of generator
                pass

            if not batch:
                break

            # Submit the batch to the executor
            for row in batch:
                future = executor.submit(process_row, row)
                futures[future] = row.name

            # Collect results as they complete
            for future in as_completed(futures):
                try:
                    result = future.result()
                    results.append(result)
                except Exception as e:
                    index = futures[future]
                    logging.error(f"Future at index {index} generated an exception: {e}")
                pbar.update(1)

            # Write the batch results to the CSV
            if results:
                df_results = pd.DataFrame(results)
                df_results.to_csv(output_file, mode='a', header=False, index=False)
                results = []

            # Clear the futures and batch for next iteration
            futures = {}
            batch = []

    pbar.close()

    logging.info(f"OCR processing completed. Results saved to {output_file}")

if __name__ == "__main__":
    main()

now this gives results but not good enough
so now your task it to improvise the whole code 
i have downloaded my images and saved it to my kaggle at /kaggle/input/the-dataset/images/test
this is how the directory looks like 
test

110EibNyclL.jpg

11TU2clswzL.jpg

11gHj8dhhrL.jpg

11lshEUmCrL.jpg

21+i52HRW4L.jpg

21-LmSmehZL.jpg

213oP6n7jtL.jpg

213wY3gUsmL.jpg



so your task it the following 
increase the accuracy of the code 
some images are repeated like 
index,image_link,group_id,entity_name
0,https://m.media-amazon.com/images/I/110EibNyclL.jpg,156839,height
1,https://m.media-amazon.com/images/I/11TU2clswzL.jpg,792578,width
2,https://m.media-amazon.com/images/I/11TU2clswzL.jpg,792578,height
3,https://m.media-amazon.com/images/I/11TU2clswzL.jpg,792578,depth
4,https://m.media-amazon.com/images/I/11gHj8dhhrL.jpg,792578,depth

in this index 1,2,3 use the same image 11TU2clswzL.jpg so when such a case occurs you need to make sure that it first checks as it is sorted by image name so repeated image will occur together so make sure it checks that as repeated images aren't downloaded again and again
also there's a catch there are some indexes omitted in the test file so when such a index comes we need it omit it from the result file as well and commas on the predicted number are not allowed if the number is 6 digit print is as 100000
if there a case where nothing is predicted output a empty string "", also once traveressed once then go check again the empty ones and try harder to find something 
i'll suggest to make a map of all allowed enetites as per constants with their abbreviations, units like for weight gram, g , kilogram, kg , etc etc, then when searching the image search of any such abbreviation as per the given on csv 
also i am running this on kaggle so make sure you use their gpus and not my cpu for fast training 
do this i hope this wil increase my accuracy
give me complete updated code