How to save the encoder layers of BERT model into array of length 738

Gaborandi: <p>i’m doing a text classification using distilbert model but before passing the encoder layers to a connected neural network , i want to save this output as pickle so i can have an array of length 738 in a sheet that i can use it later on for any task<br>
here you are my code :</p>
<p>tokenizer = DistilBertTokenizer.from_pretrained(‘distilbert-base-uncased’, do_lower_case=True)<br>
max_len = 0<br>
for sentence in text:<br>
input_ids = tokenizer.encode(sentence, add_special_tokens=True, truncation = True )<br>
max_len = max(max_len, len(input_ids))<br>
print('Max sentence length in the abstract is : ', max_len)<br>
input_ids = []<br>
attention_masks = []<br>
for sentence in text:<br>
encoded_dict = tokenizer.encode_plus(<br>
sentence,<br>
add_special_tokens = True,<br>
max_length = 64,<br>
pad_to_max_length = True,<br>
return_attention_mask = True,<br>
return_tensors = ‘pt’,<br>
truncation= True<br>
)</p>
<pre><code>input_ids.append(encoded_dict['input_ids'])
attention_masks.append(encoded_dict['attention_mask'])
</code></pre>
<p>input_ids = torch.cat(input_ids, dim=0)<br>
attention_masks = torch.cat(attention_masks, dim=0)<br>
labels = torch.tensor(labels)<br>
print('Original: ', text[0])<br>
print(‘Token IDs:’, input_ids[0])</p>
<p>from torch.utils.data import TensorDataset, random_split<br>
dataset = TensorDataset(input_ids, attention_masks, labels)<br>
train_size = int(0.8 * len(dataset))<br>
val_size = len(dataset) - train_size<br>
train_dataset, val_dataset = random_split(dataset, [train_size, val_size])</p>
<p>print(’{:&gt;5,} training samples’.format(train_size))<br>
print(’{:&gt;5,} validation samples’.format(val_size))</p>
<p>from torch.utils.data import DataLoader, RandomSampler, SequentialSampler<br>
batch_size = 16</p>
<p>train_dataloader = DataLoader(<br>
train_dataset,                          # The training samples.<br>
sampler = RandomSampler(train_dataset), # Select batches randomly<br>
batch_size = batch_size                 # Trains with this batch size.<br>
)</p>
<p>validation_dataloader = DataLoader(<br>
val_dataset,                              # The validation samples.<br>
sampler = SequentialSampler(val_dataset), # Pull out batches sequentially.<br>
batch_size = batch_size                   # Evaluate with this batch size.<br>
)<br>
from transformers import DistilBertForSequenceClassification, AdamW, DistilBertConfig<br>
model = DistilBertForSequenceClassification.from_pretrained(<br>
“distilbert-base-uncased”,                 # Use the 12-layer BERT model, with an uncased vocab.<br>
num_labels = 2,                            # The number of output labels–2 for binary classification (could be increased)<br>
output_attentions = False,                 # Whether the model returns attentions weights.<br>
output_hidden_states = False,              # Whether the model returns all hidden-states.<br>
)</p>
<p>model.to(device)</p>
<p>DistilBertForSequenceClassification(<br>
(distilbert): DistilBertModel(<br>
(embeddings): Embeddings(<br>
(word_embeddings): Embedding(30522, 768, padding_idx=0)<br>
(position_embeddings): Embedding(512, 768)<br>
(LayerNorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
)<br>
(transformer): Transformer(<br>
(layer): ModuleList(<br>
(0): TransformerBlock(<br>
(attention): MultiHeadSelfAttention(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(q_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(k_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(v_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(out_lin): Linear(in_features=768, out_features=768, bias=True)<br>
)<br>
(sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
(ffn): FFN(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(lin1): Linear(in_features=768, out_features=3072, bias=True)<br>
(lin2): Linear(in_features=3072, out_features=768, bias=True)<br>
)<br>
(output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
)<br>
(1): TransformerBlock(<br>
(attention): MultiHeadSelfAttention(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(q_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(k_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(v_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(out_lin): Linear(in_features=768, out_features=768, bias=True)<br>
)<br>
(sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
(ffn): FFN(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(lin1): Linear(in_features=768, out_features=3072, bias=True)<br>
(lin2): Linear(in_features=3072, out_features=768, bias=True)<br>
)<br>
(output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
)<br>
(2): TransformerBlock(<br>
(attention): MultiHeadSelfAttention(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(q_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(k_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(v_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(out_lin): Linear(in_features=768, out_features=768, bias=True)<br>
)<br>
(sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
(ffn): FFN(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(lin1): Linear(in_features=768, out_features=3072, bias=True)<br>
(lin2): Linear(in_features=3072, out_features=768, bias=True)<br>
)<br>
(output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
)<br>
(3): TransformerBlock(<br>
(attention): MultiHeadSelfAttention(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(q_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(k_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(v_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(out_lin): Linear(in_features=768, out_features=768, bias=True)<br>
)<br>
(sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
(ffn): FFN(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(lin1): Linear(in_features=768, out_features=3072, bias=True)<br>
(lin2): Linear(in_features=3072, out_features=768, bias=True)<br>
)<br>
(output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
)<br>
(4): TransformerBlock(<br>
(attention): MultiHeadSelfAttention(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(q_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(k_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(v_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(out_lin): Linear(in_features=768, out_features=768, bias=True)<br>
)<br>
(sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
(ffn): FFN(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(lin1): Linear(in_features=768, out_features=3072, bias=True)<br>
(lin2): Linear(in_features=3072, out_features=768, bias=True)<br>
)<br>
(output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
)<br>
(5): TransformerBlock(<br>
(attention): MultiHeadSelfAttention(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(q_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(k_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(v_lin): Linear(in_features=768, out_features=768, bias=True)<br>
(out_lin): Linear(in_features=768, out_features=768, bias=True)<br>
)<br>
(sa_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
(ffn): FFN(<br>
(dropout): Dropout(p=0.1, inplace=False)<br>
(lin1): Linear(in_features=768, out_features=3072, bias=True)<br>
(lin2): Linear(in_features=3072, out_features=768, bias=True)<br>
)<br>
(output_layer_norm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>
)<br>
)<br>
)<br>
)<br>
(pre_classifier): Linear(in_features=768, out_features=768, bias=True)<br>
(classifier): Linear(in_features=768, out_features=2, bias=True)<br>
(dropout): Dropout(p=0.2, inplace=False)<br>
)</p>
<p>i just put the last output to make the code more readable<br>
i clue how to do this task ?</p>