Looking for suggestions on creating a regression model using CNN

modeler: <p>I am using Pytorch to create a CNN for regression on synthetic data. My synthetic data are all positive.</p>
<p>It is a univariate regression problem (one output variable). The output and output were generated synthetically. The output is a gaussian distribution with mean = 1.0, and standard deviation = 0.1.</p>
<p>The input into the CNN is a 2-D tensor with 1 input channel. The 2-D tensor is 10x100. The top row of every sample is 1000, and the bottom row of every sample is 500. For the middle 8 rows, the value decreases monotonically as you move down each column, and the value increases monotonically as you move across row.</p>
<p>I am using MSE as the loss function and stochastic gradient descent for the optimization method. Are these appropriate for my data? Should I do anything differently given that my data is positive and the monotonicity of the data?</p>
<p>This is my code:</p>
<blockquote>
<p>num_epochs = 1000<br>
num_classes = 1<br>
learning_rate = 0.0000001</p>
<p>dataset, labels = synthetic.get_data()</p>
<p><span class="hashtag-raw">#convert</span> numpy array to pytorc tensor<br>
dataset = torch.tensor(dataset.astype(np.float32))<br>
labels = labels<br>
labels = torch.tensor(labels.astype(np.float32))</p>
<p>dataset = dataset.permute(2,0,1)   <span class="hashtag-raw">#permute</span> columns so 1st dimension is num_samples, 2nd is num_TCs, 3rd is num_tie_steps<br>
dataset = dataset.unsqueeze(1)  <span class="hashtag-raw">#add</span> a dimension for channels</p>
<h1>dataset = dataset.double()</h1>
<h1>labels = labels.double()</h1>
<p><span class="hashtag-raw">#divide</span> dataset into training and test<br>
<span class="hashtag-raw">#note</span> we do not need to randomly extract from dataset since there is already randomness due<br>
<span class="hashtag-raw">#to</span> how the data was generated</p>
<p>num_samples, num_input_channels, num_time_steps, num_TCs = dataset.size()<br>
num_training_samples = round(0.75*num_samples)</p>
<p>training_dataset = dataset[:num_training_samples,:,:,:]<br>
training_labels = labels[:num_training_samples]<br>
test_dataset = dataset[num_training_samples:,:,:,:]<br>
test_labels = labels[num_training_samples:]</p>
<p><span class="hashtag-raw">#function</span> for computing image width or height after convolving or pooling<br>
def calc_size(orig_size, filter_size, padding, stride, layer):<br>
length = (orig_size - filter_size + 2*padding)/stride + 1</p>
<p>if length.is_integer() is False:<br>
print(“Filter size in layer {} resulted in non-integer dimension”.format(layer))<br>
print(“This shouldn’t happen for now.”)<br>
sys.exit()</p>
<p>return int(length)</p>
<p>class ConvNet(nn.Module):<br>
def <strong>init</strong>(self):<br>
super().<strong>init</strong>()</p>
<pre><code>  filter_size_width_1 = 5
  filter_size_height_1 = filter_size_width_1
  padding_1 = 0
  stride_1 = 1

  filter_size_width_2 = 5
  filter_size_height_2 = filter_size_width_2  
  padding_2 = 0
  stride_2 = 1

  self.layer1 = nn.Sequential(
     nn.Conv2d(num_input_channels, 32, kernel_size = filter_size_width_1, stride = stride_1, padding = padding_1),      #input_channels = 1 b/c grayscale
     nn.ReLU())
     # nn.MaxPool2d(kernel_size = 2, stride = 2))

  #image starts out as num_TCs x num_time_steps (10x100)
  #now we compute the dimensions after 1st conv layer
  W = calc_size(num_TCs, filter_size_width_1, padding_1, stride_1, 1)  #=6
  H = calc_size(num_time_steps, filter_size_height_1, padding_1, stride_1, 1)  #=6

  print("w type", type(W))

  print("Width: {}, height: {} after first convolution".format(W,H))

  self.layer2 = nn.Sequential(
     nn.Conv2d(32, 64, kernel_size = filter_size_width_2, stride = stride_2),
     nn.ReLU())
     # nn.MaxPool2d(kernel_size = 2, stride = 2))

  #image starts out as num_TCs x num_time_steps (10x100)
  #now we compute the dimensions after 1st conv layer
  W = calc_size(W, filter_size_width_2, padding_2, stride_2, 2)  
  H = calc_size(H, filter_size_height_2, padding_2, stride_2, 2)

  print("Width: {}, height: {} after first convolution".format(W,H))

  self.drop_out = nn.Dropout()

  #specify 2 fully connected layers
  self.fc1 = nn.Linear(W * H * 64, 1000)
  self.fc2 = nn.Linear(1000, 1)       #10 outputs for 10 digit classes
</code></pre>
<p>def forward(self, data_to_propagate):       <span class="hashtag-raw">#analogous</span> to virtual functions in C++, we’re overriding the forward method in base class (nn.Module)<br>
out = self.layer1(data_to_propagate)<br>
out = self.layer2(out)<br>
out = out.reshape(out.size(0), -1)       <span class="hashtag-raw">#reshape</span> to 1st order tensor for FC layer<br>
out = self.drop_out(out)<br>
out = self.fc1(out)<br>
out = self.fc2(out)</p>
<pre><code>  return out
</code></pre>
<p><span class="hashtag-raw">#create</span> ConvNet instance<br>
model = ConvNet()</p>
<p><span class="hashtag-raw">#Loss</span> and optimizer<br>
cost_func = nn.MSELoss()      <span class="hashtag-raw">#this</span> contains both cross entropy and softmax<br>
optimizer = torch.optim.SGD(model.parameters(), lr = learning_rate)</p>
<p><span class="hashtag-raw">#training</span> stage<br>
loss_list = <span class="chcklst-box fa fa-square-o fa-fw"></span><br>
mse_list = <span class="chcklst-box fa fa-square-o fa-fw"></span></p>
<p>for epoch in range(num_epochs):<br>
<span class="hashtag-raw">#forward</span> pass; images var consist of batch of images<br>
outputs = model(training_dataset)<br>
print(outputs)</p>
<p>loss = cost_func(outputs, training_labels)<br>
<span class="hashtag-raw">#print</span>(“loss :”, loss)<br>
loss_list.append(loss.item())</p>
<p><span class="hashtag-raw">#backpropagation</span> and optimization<br>
optimizer.zero_grad()<br>
loss.backward()             <span class="hashtag-raw">#calculates</span> gradients in back propagation<br>
optimizer.step()            <span class="hashtag-raw">#after</span> coputing gradients, perform optimization</p>
<p><span class="hashtag-raw">#track</span> acuracy<br>
predicted = outputs.data<br>
predicted = predicted.squeeze(1)   <span class="hashtag-raw">#change</span> it from 2D 1 column array to 1D array<br>
<span class="hashtag-raw">#mse_err</span> = torch.sum((predicted.float() - training_labels) ** 2)    <span class="hashtag-raw">#output</span> of sum() is a tensor<br>
<span class="hashtag-raw">#mse_list</span>.append(mse_err)</p>
<p>print(“Epoch: {} | MSE: {}”.format(epoch, loss))</p>
</blockquote>
<p>I am getting terrible results. My predicted labels also end up negative, which shouldn’t happen. The MSE decreases, however, the convergence is very slow, and the predicted labels appear to be fluctuating. Any advice on how to remedy this?</p>

Hala_Hamdoun: <p>same what happened to me. Did you figure out the problem?</p>