Package: twomodalbert
Fine-tune a neural network with two text modes of tunable weightspip install twomodalbertIntroductionLet's consider whether we want to classify the below sample as positive or negative.contexttextThis product is horrible, which I didn't expect.Its website is just incredible!When only the message is considered, you can see its sentiment is positive due to the word 'incredible'. However, when the context is considered, sentiment will be rather negative.The TwoModalBERT package allows us to quickly run an experiment with the two-modal neural network architecture described inthis section. It allows quickly constructing a model on top ofPyTorchandtransformerslibraries and enables experimenting with weights of two input texts. So, how to use the package?UsageFirst, create aconfig.inifile in your working directory (parameters are describedhere).[GENERAL]EPOCHS=3RANDOM_SEED=42BATCH_SIZE=16MAX_SEQ_LEN=200NUM_WORKERS=2PRETRAINED_MODEL_NAME_OR_PATH=bert-base-uncasedMODEL_SAVE_PATH=best_model_state.binRead the file.fromconfigparserimportConfigParserconfig=ConfigParser()config.read("config.ini")Set the device.os.environ["CUDA_VISIBLE_DEVICES"]="0"DEVICE=torch.device("cuda"iftorch.cuda.is_available()else"cpu")Traininitialize  DataPreparation and Trainer modules.fromtwomodalbert.DataPreparationimportTwoModalDapaPreparationfromtwomodalbert.TrainerimportTwoModalBertTrainerDataPreparation=TwoModalDataPreparation(config=config)Trainer=TwoModalBertTrainer(device=DEVICE,config=config)Split inputdfwithtext,context,labelcolumns and create data loaders.(train_data_loader,train,val_data_loader,val,test_data_loader,test,)=DataPreparation.prepare_data(df,text_column="text",context_column="context",label_column="label",train_size=0.8,val_size=0.1,)Train the model (nn parameters described inthis section)model,history=Trainer.train_model(train_data_loader,train,val_data_loader,val,text_size=100,context_size=50,binary=False,text_p=0.3,context_p=0.3,output_p=0.3,)PredictLoad the model.model=TwoModalBERTModel(text_size=100,context_size=50,binary=False,text_p=0.3,context_p=0.3,output_p=0.3,)model.load_state_dict(torch.load(config["GENERAL"]["MODEL_SAVE_PATH"]))Evaluate it on a test set (choose any metric fromsklearn).y_pred,y_test=test_model(model,test_data_loader,DEVICE)Run the model on two text inputstext="Its website is just incredible!"context="This product is horrible, which I didn't expect."predict_on_text(model,text,context,DEVICE)Neural network architectureBelow you can find how TwoModalBERT is constructed and what are the class parameters.First of all, on top of the last BERT layer, a linear layer is added. To be precise, it is added on top of the CLS token. As the CLS token aggregates the entire sequence representation, it is often used in a classification task.The linear layer transforms input features with hidden size relevant to the BERT model, which is usually 768 for the models available in thetransformerspackage, into features with hidden size equal to predefinedcontext_sizeandtext_size.In the next step, the dropout layers with probabilitiescontext_pandtext_pare added on the top. Why? Because it makes the neural network less sensitive to the specific weights of neurons and not prone to overfitting.Finally, both branches created similarly are combined and followed by another dropout layer ofoutput_pand an activation function (Sigmoid ifbinary, else Softmax).Parameters inconfig.iniSettings to be defined in theconfig.inifile:VariableDescriptionDefault ValueEPOCHSthe number of complete passes through the training dataset3RANDOM_SEEDa number used to initialize a pseudorandom number generator42BARCH_SIZEthe number of training samples to work through before the model’s internal parameters are updated16MAX_SEQ_LENthe maximum length in number of tokens for the inputs to the transformer model100NUM_WORKERSthe number of processes that generate batches in parallel2PRETRAINED_MODEL_NAME_OR_PATHof a pre-trained model configuration to load from cache or download (equivalent totransformers)bert-base-uncasedMODEL_SAVE_PATHthe model save pathbest_model.binRequirenmentsconfigparser-5.3.0scikit-learn-1.0.2torch-1.12.1+cu113transformers-4.24.0