ValueError: Target and input must have the same number of elements. target nelement (20) != input nelement (40)

Xara: <p>Hi, ia m trying to run this tutorial code  <a href="https://github.com/tuhinsharma121/federated-ml/blob/master/notebooks/network-threat-detection-using-federated-learning.ipynb" rel="noopener nofollow ugc">https://github.com/tuhinsharma121/federated-ml/blob/master/notebooks/network-threat-detection-using-federated-learning.ipynb</a> on my data</p>
<blockquote>
<p>import torch</p>
<p>import torch.nn as nn</p>
<p>import torch.nn.functional as F</p>
<p>import torch.optim as optim</p>
<p>from torchvision import datasets, transforms</p>
<p>import syft as sy</p>
<p>import pandas as pd</p>
<p>colnames =[‘A’,‘B’,‘C’,‘D’]</p>
<p>df= pd.read_csv(“Z:/auction/python/PCA/data/train1.csv”, names=colnames+[‘Target’])[:7195]</p>
<p>df=df.drop(df.index[0])</p>
<p>df.head()</p>
<p>import plotly.graph_objects as go</p>
<p>from collections import Counter</p>
<p>threat_count_dict = Counter(df[‘Target’])</p>
<p>threat_types = list(threat_count_dict.keys())</p>
<p>threat_counts = [threat_count_dict[Target] for Target in threat_types]</p>
<p>print("Total distinct number of threat types : ",len(threat_types))</p>
<p>fig = go.Figure([go.Bar(x=threat_types, y=threat_counts,text=threat_counts,textposition=‘auto’)])</p>
<p><span class="hashtag-raw">#fig</span>.show()</p>
<p>numerical_colnames = [‘A’,‘B’,‘C’,‘D’]</p>
<p>final_df = df[numerical_colnames].copy()</p>
<h1>Lets remove the numerical columns with constant value</h1>
<p><span class="hashtag-raw">#numerical_df</span> = numerical_df.loc[:, (numerical_df != numerical_df.iloc[0]).any()]</p>
<h1>lets scale the values for each column from [0,1]</h1>
<h1>N.B. we dont have any negative values]</h1>
<p><span class="hashtag-raw">#final_df</span> = final_df/final_df.max()</p>
<p>X = final_df.values</p>
<h1>final dataframe has 33 features</h1>
<p>print(“Shape of feature matrix”,X.shape)</p>
<p><span class="hashtag-raw">#Construct</span> the target vector¶</p>
<p><span class="hashtag-raw">#from</span> sklearn.preprocessing import LabelEncoder</p>
<p>y = df[‘Target’].values</p>
<p><span class="hashtag-raw">#encoder</span> = LabelEncoder()</p>
<h1>use LabelEncoder to encode the threat types in numeric values</h1>
<p><span class="hashtag-raw">#y</span> = encoder.fit_transform(threat_types)</p>
<p>print("Shape of target vector : ",y.shape)</p>
<p><span class="hashtag-raw">#Train</span>/Test Split¶</p>
<p>from sklearn.model_selection import train_test_split</p>
<p>X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.4, random_state=42)</p>
<p>print("Number of records in training data : ", X_train.shape[0])</p>
<p>print("Number of records in test data : ", X_test.shape[0])</p>
<p>print("Total distinct number of threat types in training data : ",len(set(y_train)))</p>
<p>print("Total distinct number of threat types in test data : ",len(set(y_test)))</p>
<p><span class="hashtag-raw">#Lets</span> set up the environment for federated learning¶</p>
<p>#%%capture</p>
<p>import torch</p>
<p>import syft as sy</p>
<h1>Hook PyTorch ie add extra functionalities to support Federated Learning</h1>
<p>hook = sy.TorchHook(torch)</p>
<h1>Sets the seed for generating random numbers.</h1>
<p>torch.manual_seed(1)</p>
<h1>Select CPU computation, in case you want GPU use “cuda” instead</h1>
<p>device = torch.device(“cpu”)</p>
<h1>Data will be distributed among these VirtualWorkers.</h1>
<h1>Remote training of the model will happen here.</h1>
<p>gatway1 = sy.VirtualWorker(hook, id=“gatway1”)</p>
<p>gatway2 = sy.VirtualWorker(hook, id=“gatway2”)</p>
<p><span class="hashtag-raw">#Lets</span> set the training params¶</p>
<p>import numpy as np</p>
<h1>Number of times we want to iterate over whole training data</h1>
<p>BATCH_SIZE = 20</p>
<p>EPOCHS = 2</p>
<p>LOG_INTERVAL = 5</p>
<p>lr = 0.01</p>
<p>n_feature = X_train.shape[1]</p>
<p>n_class = np.unique(y_train).shape[0]</p>
<p>print("Number of training features : ",n_feature)</p>
<p>print("Number of training classes : ",n_class)</p>
<p>X_train = np.vstack(X_train[:, :]).astype(np.float)</p>
<p>X_test = np.vstack(X_test[:, :]).astype(np.float)</p>
<p>y_train = np.vstack(y_train[:]).astype(np.float)</p>
<p>y_test = np.vstack(y_test[:]).astype(np.float)</p>
<p>train_inputs = torch.tensor(X_train,dtype=torch.float).tag(“<span class="hashtag-raw">#iot</span>”, “<span class="hashtag-raw">#network</span>”,“<a class="hashtag-cooked" href="/c/data/37" data-type="category" data-slug="data" data-id="37"><span class="hashtag-icon-placeholder"><svg class="fa d-icon d-icon-square-full svg-icon svg-node"><use href="#square-full"></use></svg></span><span>data</span></a>”,“<span class="hashtag-raw">#train</span>”)</p>
<p>train_labels = torch.tensor(y_train).tag(“<span class="hashtag-raw">#iot</span>”, “<span class="hashtag-raw">#network</span>”,“<span class="hashtag-raw">#target</span>”,“<span class="hashtag-raw">#train</span>”)</p>
<p>test_inputs = torch.tensor(X_test,dtype=torch.float).tag(“<span class="hashtag-raw">#iot</span>”, “<span class="hashtag-raw">#network</span>”,“<span class="hashtag-raw">#target</span>”,“<span class="hashtag-raw">#train</span>”)</p>
<p>test_labels = torch.tensor(y_test).tag(“<span class="hashtag-raw">#iot</span>”, “<span class="hashtag-raw">#network</span>”,“<span class="hashtag-raw">#target</span>”,“<span class="hashtag-raw">#train</span>”)</p>
<h1>Send the training and test data to the gatways in equal proportion.</h1>
<p>train_idx = int(len(train_labels)/2)</p>
<p>test_idx = int(len(test_labels)/2)</p>
<p>gatway1_train_dataset = sy.BaseDataset(train_inputs[:train_idx], train_labels[:train_idx]).send(gatway1)</p>
<p>gatway2_train_dataset = sy.BaseDataset(train_inputs[train_idx:], train_labels[train_idx:]).send(gatway2)</p>
<p>gatway1_test_dataset = sy.BaseDataset(test_inputs[:test_idx], test_labels[:test_idx]).send(gatway1)</p>
<p>gatway2_test_dataset = sy.BaseDataset(test_inputs[test_idx:], test_labels[test_idx:]).send(gatway2)</p>
<h1>Create federated datasets, an extension of Pytorch TensorDataset class</h1>
<p>federated_train_dataset = sy.FederatedDataset([gatway1_train_dataset, gatway2_train_dataset])</p>
<p>federated_test_dataset = sy.FederatedDataset([gatway1_test_dataset, gatway2_test_dataset])</p>
<h1>Create federated dataloaders, an extension of Pytorch DataLoader class</h1>
<p>federated_train_loader = sy.FederatedDataLoader(federated_train_dataset, shuffle=True, batch_size=BATCH_SIZE)</p>
<p>federated_test_loader = sy.FederatedDataLoader(federated_test_dataset, shuffle=False, batch_size=BATCH_SIZE)</p>
<p>###########################<span class="hashtag-raw">#__________________________________________________________________________________________________</span></p>
<p><span class="hashtag-raw">#Lets</span> define a simple Logistic Regression Model in Pytorch</p>
<p>import torch.nn as nn</p>
<p>class Net(nn.Module):</p>
<pre><code>def __init__(self, input_dim, output_dim):

    """

    input_dim: number of input features.

    output_dim: number of labels.

    """

    super(Net, self).__init__()

    self.linear = torch.nn.Linear(input_dim, output_dim)

def forward(self, x):

    outputs = self.linear(x)

    return outputs
</code></pre>
<p>import torch.nn.functional as F</p>
<p>def train(model, device, federated_train_loader, optimizer, epoch):</p>
<pre><code>model.train()

# Iterate through each gateway's dataset

for idx, (data, tar) in enumerate(federated_train_loader):

    batch_idx = idx+1

    # Send the model to the right gateway

    model.send(data.location)

    # Move the data and target labels to the device (cpu/gpu) for computation

    #tar = tar.unsqueeze(-1) # -1 stands for last here equivalent to 1

    data, tar = data.to(device), tar.to(device)

    #tar = tar.unsqueeze(1) # -1 stands for last here equivalent to 1

    # Clear previous gradients (if they exist)

    optimizer.zero_grad()

    # Make a prediction

    output = model(data)

    

    loss = F.binary_cross_entropy(output,tar)

    

    # Calculate the gradients

    loss.backward()

    # Update the model weights

    optimizer.step()

    # Get the model back from the gateway

    model.get()

    if batch_idx==len(federated_train_loader) or (batch_idx!=0 and batch_idx % LOG_INTERVAL == 0):

        # get the loss back

        loss = loss.get()

        print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(

            epoch, batch_idx * BATCH_SIZE, len(federated_train_loader) * BATCH_SIZE,

            100. * batch_idx / len(federated_train_loader), loss.item()))
</code></pre>
<p><span class="hashtag-raw">#Lets</span> define the validation process¶</p>
<p>import torch.optim as optim</p>
<h1>Initialize the model</h1>
<p>model = Net(n_feature,n_class)</p>
<p><span class="hashtag-raw">#Initialize</span> the SGD optimizer</p>
<p>optimizer = optim.SGD(model.parameters(), lr=lr)</p>
<p>for epoch in range(1, EPOCHS + 1):</p>
<pre><code># Train on the training data in a federated way

train(model, device, federated_train_loader, optimizer, epoch)

# Check the test accuracy on unseen test data in a federated way
</code></pre>
</blockquote>
<p>I got this error<br>
RuntimeError: 1D target tensor expected, multi-target not supported<br>
so i changed<br>
loss = F.cross_entropy(output,tar) to<br>
loss = F.binary_cross_entropy(output,tar)  as i have binary classification data</p>
<p>Now i m getting this error</p>
<p>UserWarning:</p>
<p>Using a target size (torch.Size([20, 1, 1])) that is different to the input size (torch.Size([20, 2])) is deprecated. Please ensure they have the same size.</p>
<p>in binary_cross_entropy<br>
“!= input nelement ({})”.format(target.numel(), input.numel()))<br>
ValueError: Target and input must have the same number of elements. target nelement (20) != input nelement (40)</p>

ptrblck: <p>In your current script the model output contains 40 elements (<code>[20, 2]</code>) while the target contains only 20 (<code>[20, 1, 1]</code>). Assuming the target contains the class indices, you could remove the unnecessary dimensions via <code>target = target.squeeze(2).squeeze(1)</code>, make sure it’s a <code>LongTensor</code>, and use <code>nn.CrossEntropyLoss</code> instead. On the other hand, you could also one-hot encode the target and use <code>nn.BCEWithLogitsLoss</code> instead.</p>