A new neural network architecture, to boost the performance, lag the overfitting

salah55s: <p>Hello, I developed a new way of working with Neural Networks, a new feedforward, and architecture.<br>
Also a Brain-Inspired Neural Activation Module.<br>
I want to add those methods to torch modules, So please could you help me in the process of contributing or get in touch with PyTorch Staff<br>
the repository: <a href="https://github.com/salah55s/TULER.ai" class="inline-onebox" rel="noopener nofollow ugc">GitHub - salah55s/TULER.ai</a></p>
<p>thanks</p>

Cong_Chua_toc_may: <p>Using parameter based activation may reduce the information bottleneck in neural networks.</p>
<p>I think one way of going further is to double the number of weights in a neural network by 2 Siding ReLU via its forward weight connections (not backward weight connections.):</p>
<p><a href="https://ai462qqq.blogspot.com/2023/03/2-siding-relu-via-forward-projections.html" rel="noopener nofollow ugc">https://ai462qqq.blogspot.com/2023/03/2-siding-relu-via-forward-projections.html</a></p>

salah55s: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e0f5ce4b8d1e5eaf8792e4f3885508220b784cb.png" data-download-href="https://discuss.pytorch.org/uploads/default/5e0f5ce4b8d1e5eaf8792e4f3885508220b784cb" title="3"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e0f5ce4b8d1e5eaf8792e4f3885508220b784cb_2_641x500.png" alt="3" data-base62-sha1="dq5NpHuzdExHoXxEZSMlvkdFTnl" width="641" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/e/5e0f5ce4b8d1e5eaf8792e4f3885508220b784cb_2_641x500.png, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e0f5ce4b8d1e5eaf8792e4f3885508220b784cb.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/5/e/5e0f5ce4b8d1e5eaf8792e4f3885508220b784cb.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">3</span><span class="informations">702×547 52.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
this is a demo showing what is going on with each neuron, each neuron is forced to have its own activation function as shown, it always starts with very random results, but it shortly converges because of its fine-grained control " Actually there are other equations instead of</p>
<p><code>torch.where(output &gt;= self.thresholds, self.parm5 + self.parm2 * (output - self.parm4), self.parm3 * output)</code><br>
but as much as you go fast without negative values (like ReLU) as you will face lots of overfitting problems<br>
the first aim of the work was “How to overcome overfitting” and it ended up with “How to boost the ML itself”</p>
<p>And as you mentioned, such a way will be beneficial in reducing the information bottleneck in neural networks. And as you see, I was focusing on the way of the Human Brain, but it’s an amazing idea!<br>
Your insights are greatly appreciated!</p>

salah55s: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/c/8cf3812bfbe989244a510a82e81e35526781f132.png" alt="2" data-base62-sha1="k6UydWm6SPAGBtZfLL32cQ3BkGK" width="576" height="455"></p>

salah55s: <p><img src="https://discuss.pytorch.org/uploads/default/original/3X/b/8/b869e29dd3c5ce5adfd0d714d37031d94753c87d.png" alt="1" data-base62-sha1="qjoIbeY4yCRyYbBOO1LcqUJw00J" width="576" height="455"></p>

Cong_Chua_toc_may: <p>Thanks for your work and insights. I still have some information bottleneck problems with one type of neural network I experiment with, where I have to make the network wider than I would like, to compensate. I will think about your solution and try to apply it.</p>

salah55s: <p>About the information bottleneck: My work was focusing on mimicking the way of the human brain to avoid overfitting, as <strong>the brain is over-parameterized and still does not have a problem of overfitting</strong></p>
<p><strong>We may think about memorizing as an overfitting mechanism</strong>, but the human brain still has the ability to analyze well!</p>
<p>So, I think the need for a wider network is to be “smart enough” for the desired task, is because of the mimicking of the HUMAN BRAIN, as you make each neuron has powerful tools( individual prams and others) but in a very narrow scale( as each duty for each neuron is forced to do only sth with its own threshold and activation, not to have a range of outputs based on only inputs-weights. Therefore, in general, using TULERNN will make your model smarter, you may need a wider or more parameterized model and this is not a problem for now, as even with this method, you will have better performance.</p>
<p><strong>Also</strong>,<br>
<strong>The missing piece is, sth to replace the dropout</strong> “Actually it is not missing, I’m developing it to be like the inhibition mechanism in neurons, it will really different story to tell, and it is the exact point that touches the overfitting and information bottleneck problems” <strong>with.</strong></p>