subsequent calls to squad_convert_examples_to_features not executing
Author: mye4569
Pull request: no

I am new to using the huggingface transsformers and running in to issues with squad_convert_examples_to_features. Would appreciate any insights -

I have four calls to squad_convert_examples_to_features in my code(below) the first one appear to execute but the subsequent ones are skipped and the code execution randomly starts at a later point in the code, re-executes from the beginning .and hangs. Any pointers?

Code:
#... code prior to the calls
#....
#... calls to squad_convert_examples_to_features

if __name__ == '__main__':
    try:
        # Attempt to set the start method to 'spawn'
        multiprocessing.set_start_method('spawn')
    except RuntimeError:
        # If the start method has already been set, ignore the error
        pass
    
    print("Converting squad train examples to features...")
    squad_train_examples = processor.get_train_examples("file_path1", filename=None)
    squad_train_features = squad_convert_examples_to_features(
        examples=squad_train_examples, 
        tokenizer=tokenizer, 
        max_seq_length=4096, 
        doc_stride=128, 
        max_query_length=1024, 
        is_training=True,
    )
        
    print("Converting my train examples to features...")
    squad_train_examples = processor.get_train_examples(file_path2, filename="mytrain-v2.0.json")
    my_train_features = squad_convert_examples_to_features(
        examples=my_train_examples, 
        tokenizer=tokenizer, 
        max_seq_length=384, 
        doc_stride=128, 
        max_query_length=64, 
        is_training=True,
    )
        
    print("Converting squad valid examples to features...")
    squad_valid_examples = processor.get_dev_examples(file_paht3, filename=None)
    squad_valid_features = squad_convert_examples_to_features(
        examples=squad_valid_examples, 
        tokenizer=tokenizer, 
        max_seq_length=384, 
        doc_stride=128, 
        max_query_length=64, 
        is_training=False,
    )

    print("Converting my valid examples to features...")
    my_valid_examples = processor.get_dev_examples(file_path4, filename="SQuAD_validate_data.json")
    my_valid_features = squad_convert_examples_to_features(
        examples=my_valid_examples, 
        tokenizer=tokenizer, 
        max_seq_length=384, 
        doc_stride=128, 
        max_query_length=64, 
        is_training=False,
    )

print("Conversion successful!")
#....more code

Output:
Converting squad train examples to features...
100%|████████████████████████████████████████████████████████████████████████████████| 442/442 [00:50<00:00,  8.81it/s]

Comments:
Hi @mye4569, thanks for raising an issue! 

Please make sure to follow the issue template and share your running environment: run `transformers-cli env` in the terminal and copy-paste the output, as well as all relevant information to the error including the full traceback. 

It's not clear from the code being shared whether the `if name == "main"` logic is used in the script to launch this, but my first suggestion would be to try and run the code without multiprocessing. `squad_convert_examples_to_features` uses threads and there might be interactions between these two methods. 

Thanks @amyeroberts. Below is the output from running transformers-cli env. Yes, I am using the if name == "main" logic is used in the script to launch this. I removed the call to multiprocessing.set_start_method('spawn') the script and also put the calls to squad_convert_examples_to_features in a try-except block (see code below) but no exceptions are caught and the first call to squad_convert_examples_to_features just exits jumping to code outside of try-except block.


- `transformers` version: 4.38.2
- Platform: Windows-10-10.0.19045-SP0
- Python version: 3.12.2
- Huggingface_hub version: 0.21.3
- Safetensors version: 0.4.2
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.2.1+cu118 (True)
- Tensorflow version (GPU?): 2.16.0-rc0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.8.1 (cpu)
- Jax version: 0.4.25
- JaxLib version: 0.4.25
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

My Code:
#try:
        # Attempt to set the start method to 'spawn'
        # multiprocessing.set_start_method('spawn')
    #except RuntimeError:
        # If the start method has already been set, ignore the error
        #pass

    try:
        print("Getting squad train examples...")
        squad_train_examples = processor.get_train_examples("path1", filename=None)
        print("Converting squad train examples to features...")
        squad_train_features = squad_convert_examples_to_features(
            examples=squad_train_examples, 
            tokenizer=tokenizer, 
            max_seq_length=4096, 
            doc_stride=128, 
            max_query_length=1024, 
            is_training=True,
        )
            
        print("Getting my train examples to features...")
        squad_train_examples = processor.get_train_examples("path2", filename="mytrain-v2.0.json")
        print("Converting my train examples to features...")
        my_train_features = squad_convert_examples_to_features(
            examples=my_train_examples, 
            tokenizer=tokenizer, 
            max_seq_length=384, 
            doc_stride=128, 
            max_query_length=64, 
            is_training=True,
        )
            
        print("Getting squad valid examples...")
        squad_valid_examples = processor.get_dev_examples("path3", filename=None)
        print("Converting squad valid examples to features...")
        squad_valid_features = squad_convert_examples_to_features(
            examples=squad_valid_examples, 
            tokenizer=tokenizer, 
            max_seq_length=384, 
            doc_stride=128, 
            max_query_length=64, 
            is_training=False,
        )

        print("Getting my valid examples...")
        my_valid_examples = processor.get_dev_examples("path4", filename="SQuAD_validate_data.json")
        print("Converting my valid examples to features...")
        my_valid_features = squad_convert_examples_to_features(
            examples=my_valid_examples, 
            tokenizer=tokenizer, 
            max_seq_length=384, 
            doc_stride=128, 
            max_query_length=64, 
            is_training=False,
        )
    except Exception as e: 
        print("exception...")
        print (e)

print("Conversion successful!")#try:


Below is the runtime output - the squad_convert_examples_to_features fails at 40% conversion with a memory error - any suggestion,

Getting training and validate examples...
Getting squad train examples...
100%|████████████████████████████████████████████████████████████████████████████████| 442/442 [00:32<00:00, 13.46it/s]
Getting my train examples to features...
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 25.79it/s]
Getting squad valid examples...
100%|██████████████████████████████████████████████████████████████████████████████████| 35/35 [00:04<00:00,  8.56it/s]
Getting my valid examples...
100%|████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00, 24.21it/s]
Converting squad train examples to features...
convert squad examples to features:  40%|███████████████                       | 51584/130319 [07:22<11:15, 116.61it/s]
exception...
Error sending result: '[[<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63EC0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63B30>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62660>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62780>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63FE0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63CE0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63D10>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62720>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63C50>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63920>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62900>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62BD0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63350>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63B00>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63C80>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62390>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62C90>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62B40>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED632F0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62960>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED619D0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED620F0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62FF0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED61970>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED61A90>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED61B50>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED631A0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63530>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED63380>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED62A80>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED61BB0>], [<transformers.data.processors.squad.SquadFeatures object at 0x000002608ED61B80>]]'. Reason: 'MemoryError()'

@mye4569 Thanks for sharing more details. 

Based on these logs, it looks like you're hitting a memory error. I'd suggest looking at your CPU utilisation to see whether you're hitting OOM. 

I'd suggest trying to run just one of the `squad_convert_examples_to_features` calls on its own, and see whether that is successful

Thanks again @amyeroberts. If I do not call squad_convert_examples_to_features on SQUAD training and validation data and just call it for my training and validation data then i do not get this error. My machine has 8.00 GB (7.85 GB usable) and is a Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz   2.60 GHz - 64-bit operating system, x64-based processor. I just want to use the BertForQuestionAnswering model to train it on my domain specific training and validation data. I think this requires that the model be trained on a features that are  a combination of SQUAD and my training features - am I missing/incorrect something in my approach? Is there anyway to get this to work? Thank you.

Hi @mye4569, 

Without having access to either of the datasets, it's hard to know how to make this work for you. As it's a memory error, I'd try to look into ways to reduce this e.g. reducing the `doc_stride` argument. 

The method being used here is old ~4 years, and so I don't have a great knowledge of how it's supposed to be used. 

There are legacy (not maintained) examples of finetuning BERT on squad here:
https://github.com/huggingface/transformers/tree/main/examples/legacy/question-answering

There are maintained examples of fine-tuning question answering models on datasets like squad here: 
https://github.com/huggingface/transformers/tree/main/examples/pytorch/question-answering

Both should show you the general approach for doing this in transformers. I'd recommend looking at the maintained examples first. 

General questions like this are best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.