Non-sliding kernels for location-aware processing in Convolutional Neural Networks

vigi: <p>My understanding of how CNN operates in image detection is through the use of kernels that slide through the image to detect features (edges and so on). So a single kernel could potentially be learning to detect an edge no matter where it is in the image. This is great for image recognition problems where an image of a dog shifted to the right or inverted is still an image of a dog. <a href="https://towardsdatascience.com/intuitively-understanding-convolutions-for-deep-learning-1f6f42faee1" rel="noopener nofollow ugc">This article</a> states “the features the kernel learns must be general enough to come from any part of the image”. The article also states how using CNN for categorical data where the order in which data is organised is irrelevant can be “disastrous”.</p>
<p>However, there are instances where it is desirable for the algorithm to be location-aware in order to classify better. Take the case of using CNN to train a network that will predict card play in the game of bridge (a version of double-dummy where all cards are laid out open - perfect information, deterministic). At the beginning of the game the cards dealt to the four could look (very unrealistically) something like this.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/a/bae958b1d6d6117b48b219cccfe4f63866f69b36.png" data-download-href="https://discuss.pytorch.org/uploads/default/bae958b1d6d6117b48b219cccfe4f63866f69b36" title="Screenshot 2021-12-01 at 11.03.33"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/bae958b1d6d6117b48b219cccfe4f63866f69b36_2_690x260.png" alt="Screenshot 2021-12-01 at 11.03.33" data-base62-sha1="qFuKCFJ2SWTVKWtPaMDLa8SAC5E" width="690" height="260" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/bae958b1d6d6117b48b219cccfe4f63866f69b36_2_690x260.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/bae958b1d6d6117b48b219cccfe4f63866f69b36_2_1035x390.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/a/bae958b1d6d6117b48b219cccfe4f63866f69b36_2_1380x520.png 2x" data-dominant-color="F6F0EF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-12-01 at 11.03.33</span><span class="informations">2726×1030 160 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>where Leader = the player playing the lead card in round 1, and the subsequent players organised as Leader.LeftHandOpponent, Leader.Partner and Leader.RightHandOpponent. Each player’s cards are organised in four suits starting from the Trump_Suit and then the other suits in the original suit hierarchy. Cards go from highest value in the top ‘A’ to lowest value in the bottom ‘2’.</p>
<p>Here is a transpose of the image above.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/4/54a2a346d2b7c8dfa0e65e3140324f2684bb767d.png" data-download-href="https://discuss.pytorch.org/uploads/default/54a2a346d2b7c8dfa0e65e3140324f2684bb767d" title="Screenshot 2021-12-01 at 11.17.18"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54a2a346d2b7c8dfa0e65e3140324f2684bb767d_2_519x500.png" alt="Screenshot 2021-12-01 at 11.17.18" data-base62-sha1="c4IyyD3UIydBGWXOe90lYXEvAOF" width="519" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54a2a346d2b7c8dfa0e65e3140324f2684bb767d_2_519x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54a2a346d2b7c8dfa0e65e3140324f2684bb767d_2_778x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/4/54a2a346d2b7c8dfa0e65e3140324f2684bb767d_2_1038x1000.png 2x" data-dominant-color="F0E9E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-12-01 at 11.17.18</span><span class="informations">1186×1142 85.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This layout provides a lot of visual cues in terms of how the gameplay will proceed and who will end up winning how many tricks if viewed it from the perspective of control cards distribution within each suit and hand strength. So, the answer to the question of will CNN actually be able to process this data to provide good predictions is a resounding Yes (at least to me).</p>
<p>However, here is the problem - A regular CNN with a sliding kernel with a (4, 1) stride and no padding would make no distinction between the red boxes when in reality there is a massive difference between them.</p>
<p>Possible Solution? - A filter consisting of non-sliding kernels/kernels that only slide in one direction (perhaps horizontally or vertically) however would theoretically only seek to learn location-aware features and that could potentially improve accuracy? Just shooting arrow in the sky.</p>
<p>Has this been researched? Has anybody implemented this already? Could this work?</p>
<p>P.S: CNN has been used on AlphaGo Zero was great success. Obviously in the game of Go, patterns located in the top of the board carry the same weight as those located in the bottom. The gameplay does not change if the board is flipped 180 degrees. This however is not the case in the game of contract bridge. I am looking at ideas of how this can be resolved.</p>

my3bikaht: <p>Just a food for thought: <a href="https://arxiv.org/abs/2108.07884" class="inline-onebox" rel="noopener nofollow ugc">[2108.07884] Global Pooling, More than Meets the Eye: Position Information is Encoded Channel-Wise in CNNs</a></p>

vigi: <p>Thank you for sharing this. If my understanding is correct, the research suggests that global pooling retains channel information while spatial information is collapsed (which is a bit counterintuitive to me). They further propose measures to improve translation invariance, which is the problem I am trying to get rid of.</p>
<p>In my example above on Bridge, what I am hoping to achieve is ‘translation equivariance’ along the horizontal direction if data is structured as it is image 1. This way, the output shall change if the red box in the first three rows move to the right - which is desirable.</p>
<p>I am also hoping to achieve ‘translation varaince’ along the vertical direction if the input data is structured as it is in the first image. This way, there is a clear distinction being made between the first, second, third, and fourth red boxes.</p>