Merging different size vectors in fully connected layers

maryma: <p><strong>Hi everyone. I needed your help and advice on my problem. I want to do a survival prediction based on CT images and two clinical variables of patients(age and gender). For CT images feature representation I tried to do a classification and I extracted the features before fully connected layers the feature vector for every patient has a size of 2560. So for the second network, I have to give age, gender, and feature vector to do the survival estimation. I don’t know how to merge the feature vector and the two variables. One way can be the concatenation but I think because of the big size of the image vector the age and gender wouldn’t affect the prediction. Another way can be using some fully connected layers to decrease the image feature vector size and then add age and gender to the output of that FC layer and have other FC layers to predict the survival. But the problem is that when we increase the number of FC layers the accuracy decreases. Do you have any idea how it is possible to merge a feature vector and two variables in a way that gets a good result and accuracy?</strong><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a7ada3051a65da9eccd545b80a382b53d106a92.png" data-download-href="https://discuss.pytorch.org/uploads/default/7a7ada3051a65da9eccd545b80a382b53d106a92" title="Screenshot from 2023-01-24 13-22-20"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a7ada3051a65da9eccd545b80a382b53d106a92_2_690x381.png" alt="Screenshot from 2023-01-24 13-22-20" data-base62-sha1="htvsAYMiVDhjKRGOatzOyhbR08W" width="690" height="381" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/a/7a7ada3051a65da9eccd545b80a382b53d106a92_2_690x381.png, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a7ada3051a65da9eccd545b80a382b53d106a92.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/a/7a7ada3051a65da9eccd545b80a382b53d106a92.png 2x" data-dominant-color="F6F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2023-01-24 13-22-20</span><span class="informations">718×397 22.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="maryma" data-post="1" data-topic="170963" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryma/48/56556_2.png" class="avatar"> maryma:</div>
<blockquote>
<p><strong>Do you have any idea how it is possible to merge a feature vector and two variables in a way that gets a good result and accuracy?</strong></p>
</blockquote>
</aside>
<p>My intuition suggests that you would want to simply concatenate the feature<br>
vector with the two additional variables and let training of the network sort<br>
out how exactly to mix the values together.</p>
<p>I do see your point that your additional two variables may be overwhelmed<br>
by the 2560 values in your feature vector.  However, that’s the basic reality<br>
of it – your CT images contain much more information than the two age and<br>
gender variables.  So it makes sense that the feature vector derived from the<br>
CT images should have much more effect on the final predictions than age<br>
and gender.</p>
<p>I also think that you would want to provide your concatenated vector as input<br>
to your second-to-last (or maybe even your third-to-last) fully-connected layer,<br>
rather than as input to your final fully-connected layer.  The idea is that you<br>
want age and gender to get mixed nonlinearly with the feature vector by the<br>
nonlinear activation between your second-to-last and final layers.  The<br>
nonlinear activations – this nonlinear mixing – are how the inputs to your<br>
network “communicate” with one another and are the key to how neural<br>
networks work.</p>
<p>If your current network only has a single fully-connected layer at the end,<br>
then I would seriously consider adding a second fully-connected layer at<br>
the end so you can get the benefit of the nonlinear activation in between<br>
the two.  If you’re concerned that adding the additional layer could degrade<br>
performance – and it might well in the absence of the age and gender<br>
variables – try both with and without the second layer to see whether the<br>
benefits of the nonlinear activation outweigh the cost of the additional<br>
layer.</p>
<p>Best.</p>
<p>K. Frank</p>

maryma: <p>Thanks for your reply.</p>
<p>It is true that the CT images have more information than age and gender but it doesn’t have more effect on the output(prediction).</p>
<p>So I was thinking to make gender and age a vector of 2560 by replicating the values and then concatenating them with CT images and give the concatenated vector to the fully connected layers. But I am not sure whether it can be the correct method or not.</p>
<p>Also, I can use age and gender in the middle layers as you told me to use the non-linearity function to compare two ways.</p>

KFrank: <p>Hi Maryam!</p>
<aside class="quote no-group" data-username="maryma" data-post="3" data-topic="170963" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maryma/48/56556_2.png" class="avatar"> maryma:</div>
<blockquote>
<p>So I was thinking to make gender and age a vector of 2560 by replicating the values</p>
</blockquote>
</aside>
<p>My intuition suggests that replicating the age and gender values (thousands<br>
of times!) is likely to do more harm than good.</p>
<p>My thinking is that you would have all of these variables that are actually<br>
the same and the network would have to “learn” that these variables are<br>
just copies of one another.  I believe that it would make more sense to just<br>
concatenate the two values (unreplicated) to the feature vector and let the<br>
final layer(s) “learn” that your concatenated vector of length 2562 consists<br>
of 2560 features and the two additional age and gender values.  This is<br>
very much the kind of thing that neural networks can learn.</p>
<p>Of course, if you can afford the training time, you could try it both ways, but<br>
I expect that not replicating will work better.</p>
<p>Best.</p>
<p>K. Frank</p>