text: 
  
  TITLE
  BertModel for 2 category classification - How to evaluate the performance
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  Hello there, I am building a fine-tuned BERT model for classification (with a linear layer in the end). The prediction should just be 1/0 (Yes, No). 
  
  When I am writing the evaluation part, I saw some people online did a F.log_softmax for the logits then use np.argmax to get the predicted label. However, I also saw people directly apply np.argmax on logits without the softmax activation. I am wondering which one should I follow and how to decide that? 
  
  Here's my model definition:
  ```python
  class ReviewClassification(BertPreTrainedModel):
      def __init__(self, config):
          super().__init__(config)
          self.num_labels = 2
  
          self.bert = BertModel(config)
          self.dropout = nn.Dropout(config.hidden_dropout_prob)
  
          embedding_size = config.hidden_size
  
          self.classifier = nn.Linear(embedding_size, len(LABEL_NAME))
          self.init_weights()
  
      def forward(
              self,
              review_input_ids=None,
              review_attention_mask=None,
              review_token_type_ids=None,
              agent_input_ids=None,
              agent_attention_mask=None,
              agent_token_type_ids=None,
              labels=None,
      ):
  
          review_outputs = self.bert(
              review_input_ids,
              attention_mask=review_attention_mask,
              token_type_ids=review_token_type_ids,
              position_ids=None,
              head_mask=None,
              inputs_embeds=None,
          )
          
          feature = review_outputs[1]  # (batch_size, seq_len) -? Should it be (batch_size, hidden_size)
  
          # nn.CrossEntropyLoss applies F.log_softmax and nn.NLLLoss internally on your input,
          # so you should pass the raw logits to it.
          logits = self.classifier(feature)
  
          outputs = (logits,)  # + outputs[2:]  # add hidden states and attention if they are here
  
          if labels is not None:
              loss_fct = nn.CrossEntropyLoss()
              loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
              outputs = (loss,) + outputs
  
          return outputs  # (loss, logits, hidden_states, attentions) 
  ```
  
  Then this is my validation code
  ```
  def model_validate(model, data_loader):
      # Put the model in evaluation mode--the dropout layers behave differently
      # during evaluation.
      model.eval()
      device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
      model.to(device)
      if torch.cuda.device_count() > 1:
          model = nn.DataParallel(model)
  
      label_prop = data_loader.dataset.dataset.label_prop()
  
      total_valid_loss = 0
  
      batch_size = data_loader.batch_size
      num_batch = len(data_loader)
  
      y_pred, y_true = [], []
  
      # Evaluate data
      for step, batch in tqdm(enumerate(data_loader), desc="Validation...", total=num_batch):
          b_review_input_ids = batch["review_input_ids"].to(device)
          b_review_attention_mask = batch["review_attention_mask"].to(device)
          b_review_token_type_ids = batch["review_token_type_ids"].to(device)
  
          b_binarized_label = batch["binarized_label"].to(device)
  
          # Tell pytorch not to bother with constructing the compute graph during
          # the forward pass, since this is only needed for backprop (training).
          with torch.no_grad():
              (loss, logits,) = model(review_input_ids=b_review_input_ids,
                                      review_attention_mask=b_review_attention_mask,
                                      review_token_type_ids=b_review_token_type_ids,
  
                                      labels=b_binarized_label)
  
          total_valid_loss += loss.item()
          numpy_probas = logits.detach().cpu().numpy()
          y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
          y_true.extend(b_binarized_label.cpu().numpy())
      # End of an epoch of validation
  
      # put model to train mode again.
      model.train()
  
      ave_loss = total_valid_loss / (num_batch * batch_size)
  
      # compute the various f1 score for each label
      report = classification_report(y_true, y_pred, output_dict=True)
      metrics_df = pd.DataFrame(report).transpose()
     
  
      metrics_df = metrics_df.sort_index()
      
      weighted_f1_score = metrics_df.loc['weighted avg', 'f1-score']
      averaged_f1_score = metrics_df.loc['macro avg', 'f1-score']
  
      return ave_loss, metrics_df, {
          "weighted": weighted_f1_score,
          "averaged": averaged_f1_score,
      }
  ```
  
  The other version I was trying is:
  ```
      transfored_logits = F.log_softmax(logits,dim=1)
      numpy_probas = transfored_logits.detach().cpu().numpy()
      y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
      y_true.extend(b_binarized_label.cpu().numpy())
  
  ```
  
  The third version I was trying is:
  ```
     transfored_logits = torch.sigmoid(logits)
      numpy_probas = transfored_logits.detach().cpu().numpy()
      y_pred.extend(np.argmax(numpy_probas, axis=1).flatten())
      y_true.extend(b_binarized_label.cpu().numpy())
  
  ```
  
  I also don't know how to understand the result. When I see online, people say if I set dim = 1 for log_softmax, the sum of probability for all feature (categories) should = 1. However, giving some examples below:
  
  This is logits output: (for one batch - batch size = 16, num_labels = 2)
  tensor([[ 1.1261, -1.8547],
          [ 0.6066, -1.1498],
          [ 1.3667, -2.0078],
          [ 2.0652, -2.6669],
          [ 1.0388, -1.7555],
          [ 0.6801, -1.1652],
          [ 0.8315, -1.3860],
          [ 1.5685, -2.2362],
          [ 0.1150, -0.3344],
          [ 2.0751, -2.6166],
          [ 1.5033, -2.1702],
          [ 0.1115, -0.3096],
          [ 0.8610, -1.4834],
          [ 1.5544, -2.2773],
          [ 2.1014, -2.6533],
          [ 0.7789, -1.3748]], device='cuda:0')
  
  If I apply softmax first, F.log_softmax(logits,dim=1), I get: 
  tensor([[-0.0495, -3.0302],
          [-0.1593, -1.9157],
          [-0.0337, -3.4082],
          [-0.0088, -4.7409],
          [-0.0594, -2.8537],
          [-0.1467, -1.9920],
          [-0.1033, -2.3209],
          [-0.0220, -3.8267],
          [-0.4935, -0.9429],
          [-0.0091, -4.7008],
          [-0.0251, -3.6985],
          [-0.5046, -0.9257],
          [-0.0916, -2.4360],
          [-0.0214, -3.8531],
          [-0.0086, -4.7632],
          [-0.1098, -2.2635]], device='cuda:0')
  
  The sum per row doesn't sum up to 1 and doesn't look like probability to me. 
  
  If I apply sigmoid, torch.sigmoid(logits)
  tensor([[0.7551, 0.1353],
          [0.6472, 0.2405],
          [0.7969, 0.1184],
          [0.8875, 0.0650],
          [0.7386, 0.1474],
          [0.6638, 0.2377],
          [0.6967, 0.2000],
          [0.8276, 0.0965],
          [0.5287, 0.4172],
          [0.8885, 0.0681],
          [0.8181, 0.1025],
          [0.5278, 0.4232],
          [0.7029, 0.1849],
          [0.8255, 0.0930],
          [0.8910, 0.0658],
          [0.6854, 0.2018]], device='cuda:0')
  
  It does look like probability more, although it still doesn't sum up to 1. 
  
  No matter which version I use, the predicted result is always the same in this case (since my 1 (Yes) Label has a really low incidence rate)
  array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0])
  
  