Transfer learning (or fine-tuning) pre-trained model on non-text data

username: corvusMidnight
cooked: <p>I am currently fine-tuning a <a href="https://huggingface.co/unideeplearning/polibert_sa" rel="noopener nofollow ugc">sentiment analysis bert-based model</a> using <a href="https://huggingface.co/docs/transformers/training#train-with-pytorch-trainer" rel="noopener nofollow ugc">PyTorch Trainer</a> from hugging face. So far, so good.</p>
<p>I have easily managed to fine-tune the model on my text data. However, I’d like to conduct an ablation study to see how the inclusion of linguistic features impacts the model’s performance.</p>
<p>In other words, how the inclusion of, e.g., comment length, type-to-token ratio, and other features (stored in my dataset in a separate column) affects the performance of my model.</p>
<p>This is what my data kind of looks like:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>Text</th>
<th>Type-token ratio</th>
</tr>
</thead>
<tbody>
<tr>
<td>Hi, my name is…</td>
<td>1.0</td>
</tr>
<tr>
<td>I cannot believe I did not…</td>
<td>0.95</td>
</tr>
</tbody>
</table>
</div><p>In the specific case above, for instance, I would like to fine-tune the model on the text column but also on the Type-token ratio one.</p>
<p>I know that some people concatenate the two columns into a string, but I am not sure that is the correct method. Is there a more methodologically sound way of doing this?</p>
<p>I was not able to find much information about it.</p>
<p>This is my code:</p>
<pre><code>dataset = pd.read_csv('/content/gdrive/MyDrive/.../data_train.csv')
train_roberta = dataset[['text_lower', 'label']].sample(frac=0.75)
validation_roberta = dataset[['text_lower', 'label']].drop(train_roberta.index)
train_roberta = train_roberta.dropna()
validation_roberta = validation_roberta.dropna()
train = Dataset.from_pandas(train_roberta, preserve_index=False)
validation = Dataset.from_pandas(validation_roberta, preserve_index=False)

tokenizer = AutoTokenizer.from_pretrained("a_model/a_bert_like_model")

def tokenize_function(example):
    return tokenizer(example["text_lower"], padding="max_length", truncation=True)

tokenized_train_dataset = train.map(tokenize_function, batched=True)
tokenized_test_dataset = validation.map(tokenize_function, batched=True)
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
import os
os.environ["WANDB_DISABLED"] = "true"
training_args = TrainingArguments("test-trainer", evaluation_strategy="epoch")   # default arguments for fine-tuning
model = AutoModelForSequenceClassification.from_pretrained("a_model/a_bert_like_model", num_labels=3)


def compute_metrics(eval_preds):   # compute accuracy and f1-score
    metric = load_metric("glue", "mrpc")
    logits, labels = eval_preds
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

trainer = Trainer(   # specifying trainer class
    model,
    training_args,
    train_dataset=tokenized_train_dataset,
    eval_dataset=tokenized_test_dataset,
    data_collator=data_collator,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

trainer.train()  # starts fine-tuning
</code></pre>