Insights from computer game to create construction, destruction based neural network to make Siri learn like human

vainaijr: <p>In order to make Siri learn like human learn isn’t the best way be to see from how human learn to play computer game?</p>
<p>When human play computer game, then in the beginning nothing learnt, random gameplay, overtime what happen is, that if one gets a positive feedback / construction / success, then the technique be learnt, while if one gets a negative feedback  / destruction / failure, then the technique be changed, the word technique here would mean a combination of keys, like move forward -&gt; jump -&gt; dive …</p>
<p>Another thing be that it take time to change technique, for example if one learns a particular combination of keybinds for a move, and then one day decides to change the keybinds, then one is not able to play with the new keybinds immediately, it takes time to update a technique, and for some duration one would keep using the same old keybinds.</p>
<p>Another thing be that one would not consider the real world facts in a game, for example if asked do you want to eat this candy, then in the game world, the game developer could program the rules, that your health would be negative if you eat candy, so one would avoid eating candy in game, as it has destructive output, while in the real world, eating a candy would give good taste, that is a constructive outcome, so, one would do it in the real world, that is our actions are based on this notion of success / failure.</p>
<p>One more insight be that, on a positive feedback, the combination of keys only get strongly learnt, like, one get used to a particular combination of keys, if they have had a positive feedback, using the same combination multiple times, that is if a particular set of moves was X-&gt;Y-&gt;Z, then this combination would be strongly learnt, while on a negative feedback, it lead to combination of keys, getting changed, like, if one used to have a particular set of moves X-&gt;Y-&gt;Z, and it lead to failure, then they would update it to something like X-&gt;Z-&gt;Y, or break certain combination like X-&gt;Z.</p>
<p>Is there a term for this type of neural network?</p>

ptrblck: <p>I’m not an expert in Reinforcement Learning, but your description seems to point towards this field of machine learning, which uses rewards as a training signal and thus can be used to learn how to play video games.</p>
<p>One interesting thing in my opinion is the creation of rewards.<br>
Let’s stick to video games, since you’ve mentioned them. While some “moves” might give a negative reward after their execution (you might lose life or points in any sense), they might be beneficial in the long run and you might even win the game with them.<br>
Also, it’s always funny to watch when some agents are focusing on the “wrong” rewards while playing games, such as here:</p><div class="youtube-onebox lazy-video-container" data-video-id="tlOIHko8ySg" data-video-title="CoastRunners 7" data-provider-name="youtube">
  <a href="https://www.youtube.com/watch?v=tlOIHko8ySg" target="_blank" rel="noopener">
    <img class="youtube-thumbnail" src="https://img.youtube.com/vi/tlOIHko8ySg/hqdefault.jpg" title="CoastRunners 7" width="480" height="360">
  </a>
</div>

<p>The learned movement of grabbing these “turbo” boxes gave apparently more points than winning the game and was thus considered to be better. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"></p>
<p>This is also an interesting article from OpenAI:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://openai.com/research/emergent-tool-use">
  <header class="source">

      <a href="https://openai.com/research/emergent-tool-use" target="_blank" rel="noopener">openai.com</a>
  </header>

  <article class="onebox-body">
    <img src="https://openaicom.imgix.net/40c42048-863a-4651-862a-223a15c21cbf/emergent-tool-use.jpg?auto=compress%2Cformat&amp;fit=min&amp;fm=jpg&amp;q=80&amp;rect=0%2C442%2C2659%2C2664" class="thumbnail onebox-avatar" width="499" height="499">

<h3><a href="https://openai.com/research/emergent-tool-use" target="_blank" rel="noopener">Emergent tool use from multi-agent interaction</a></h3>

  <p>We’ve observed agents discovering progressively more complex tool use while playing a simple game of hide-and-seek. Through training in our new simulated hide-and-seek environment, agents build a series of six distinct strategies and...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


vainaijr: <p>Furthermore, another insight be that, our actions are always based on this notion of success / failure, the example that you share, where the player, goes for the ‘wrong’ rewards, is still going for success.</p>
<p>Consider a person who has played, a game 100 times, and in each of the 100 times they played that game, they won, they are so much used to winning that they are bored.<br>
For the first 100 games, the notion of success, always meant winning.<br>
that is,</p>
<pre><code class="lang-auto">first 100 games -&gt; success -&gt; win the game
</code></pre>
<p>In the 101st game, the person is bored, now, they decide to create a funny clip / a meme, and share it with others, now the notion of success, changes from winning the game to creating a funny clip, that is,</p>
<pre><code class="lang-auto">101st game -&gt; success -&gt; create a funny clip
</code></pre>
<p>or another scenario could be that, in the 102nd game, the person says that now I would want to give my friend a win (assuming it be a multi player game, and their friend is one of the opponents), so the notion of success, would change from winning a game, to giving friend a win, that is,</p>
<pre><code class="lang-auto">102nd game -&gt; success -&gt; give friend a win
</code></pre>
<p>but in every game, the person is still going for success, for example, in the 101st game, they would consider the scenario of winning, and not being able to create a funny clip as a failure, that is,</p>
<pre><code class="lang-auto">101st game -&gt; failure -&gt; no funny clip, but won
</code></pre>
<p>or in the 102nd game, they would consider it a failure, if their friend lost, that is,</p>
<pre><code class="lang-auto">102nd game -&gt; failure -&gt; friend lost
</code></pre>
<p>In the 103rd game, the person might set the notion of success, to be the first person to get eliminated, and if they survive longer than anyone else, then that would be a failure, that is,</p>
<pre><code class="lang-auto">103rd game -&gt; success -&gt; first person to get eliminated
</code></pre>
<pre><code class="lang-auto">103rd game -&gt; failure -&gt; survived longer than others
</code></pre>
<p>but, in the all of the games, the person attempts at success, what they consider as success keeps changing.</p>

vainaijr: <p>Furthermore, another insight be that, one word on its own does not have a meaning, in order for a word to have a meaning, there must be some more words (or image).<br>
But in the current state, the embedding based approach in text experiments, keep updating the so called representation of each word, as if it has nothing to do with any other word, and as if each word be isolated on its own, I think this be a big problem, which do not consider that words be connected to each other.</p>

ptrblck: <p>I’m always reading your interesting thoughts, but have to be a bit picky regarding the medical imaging claims.<br>
Even today CT and MRT scans are displayed as black and white images on special monitors, which can “properly” display all necessary levels of gray. That being said, you can find colored images, e.g. via functional MRT images, where the color might denote the flow direction or speed of the liquid (blood) in the vessel. While color codes can be helpful (as seen in the fMRT example) often a grayscale image (on a medical monitor) is more beneficial and not necessarily limited by the compute power your machine has. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"><br>
The most important aspect is what exactly do you want and need to see in the image.<br>
Medical software usually comes with some presets using special “windows” to display certain tissue classes. E.g. if you are only interested in the lungs, you would select the “lungs window” so that the proper voxel intensities will be scaled to the full monitor range.</p>