AttnRNN: An Efficient Recurrent Neural Network Model Based on Attention 

username: wqws
cooked: <h1><a name="p-472418-attnrnn-an-efficient-recurrent-neural-network-model-based-on-attention-mechanism-1" class="anchor" href="#p-472418-attnrnn-an-efficient-recurrent-neural-network-model-based-on-attention-mechanism-1"></a>AttnRNN: An Efficient Recurrent Neural Network Model Based on Attention Mechanism</h1>
<h2><a name="p-472418-abstract-2" class="anchor" href="#p-472418-abstract-2"></a>Abstract</h2>
<p>This paper proposes a novel recurrent neural network architecture—<strong>AttnRNN</strong>—which innovatively combines attention mechanisms with gating mechanisms to address the information retention problem in traditional RNNs for long-sequence tasks. Experiments demonstrate that the model <strong>significantly outperforms</strong> standard RNN, GRU, and LSTM models on the Adding Problem, Copy Memory task, and IMDB text classification, while maintaining low parameter count and computational complexity.</p>
<h2><a name="p-472418-h-1-model-architecture-3" class="anchor" href="#p-472418-h-1-model-architecture-3"></a>1. Model Architecture</h2>
<h3><a name="p-472418-h-11-core-components-4" class="anchor" href="#p-472418-h-11-core-components-4"></a>1.1 Core Components</h3>
<pre data-code-wrap="python"><code class="lang-python">class EfficientAttention(nn.Module):
    def __init__(self, embed_dim, num_heads):
        # Project only Q-values, KV uses raw context
        self.q_proj = nn.Linear(embed_dim, embed_dim)
        
    def forward(self, query, context):
        # Multi-head attention calculation (no KV projection)
        ...

class AttnRNNCell(nn.Module):
    def __init__(self, input_size, hidden_size):
        # Context construction: Combination of h_prev and x
        self.context = torch.cat([h_exp, x_exp, h_exp+x_exp, h_exp*x_exp], dim=1)
        
        # Attention mechanism
        self.attn = EfficientAttention(hidden_size, num_heads)
        
        # Single-gate design
        self.update_gate = nn.Sequential(
            nn.Linear(2*hidden_size, hidden_size),
            nn.Sigmoid()
        )
        
        # Residual connection
        self.layer_norm = nn.LayerNorm(hidden_size)
        
    def forward(self, h_prev, x):
        # Attention calculation → Gated fusion → Residual update
        ...
</code></pre>
<h3><a name="p-472418-h-12-design-innovations-5" class="anchor" href="#p-472418-h-12-design-innovations-5"></a>1.2 Design Innovations</h3>
<ul>
<li><strong>Attention Competition Mechanism</strong>: Direct competition for attention scores between historical information (h_prev) and new inputs (x) within a fixed-length context (4 tokens)</li>
<li><strong>Residual Dominance Principle</strong>: Residual connections ensure historical information always dominates</li>
<li><strong>Minimalist Gating Design</strong>: Single-gate structure (vs. GRU’s 3 gates/LSTM’s 4 gates) reduces parameters</li>
<li><strong>No KV Projection</strong>: Attention module projects only Q-values, significantly lowering computation</li>
</ul>
<h2><a name="p-472418-h-2-theoretical-analysis-6" class="anchor" href="#p-472418-h-2-theoretical-analysis-6"></a>2. Theoretical Analysis</h2>
<h3><a name="p-472418-h-21-parameter-comparison-7" class="anchor" href="#p-472418-h-21-parameter-comparison-7"></a>2.1 Parameter Comparison</h3>
<div class="md-table">
<table>
<thead>
<tr>
<th>Model</th>
<th>Parameter Formula (d:input dim, h:hidden dim)</th>
<th>d=64, h=128 Example</th>
</tr>
</thead>
<tbody>
<tr>
<td>Vanilla RNN</td>
<td>dh + h² + 2h</td>
<td>24,832</td>
</tr>
<tr>
<td>GRU</td>
<td>3(dh + h² + h)</td>
<td>74,496</td>
</tr>
<tr>
<td>LSTM</td>
<td>4(dh + h² + h)</td>
<td>99,328</td>
</tr>
<tr>
<td><strong>AttnRNN</strong></td>
<td>dh + 4h² + 6h</td>
<td><strong>66,432</strong></td>
</tr>
</tbody>
</table>
</div><p><strong>Parameter Distribution</strong>:</p>
<ol>
<li>Data alignment projection layer: dh</li>
<li>Attention module: 3h² (q_proj + out_proj)</li>
<li>Gating layer: h² + h</li>
<li>Normalization layer: 2h</li>
</ol>
<h3><a name="p-472418-h-22-time-complexity-8" class="anchor" href="#p-472418-h-22-time-complexity-8"></a>2.2 Time Complexity</h3>
<div class="md-table">
<table>
<thead>
<tr>
<th>Model</th>
<th>Per-step Complexity</th>
<th>Sequence Complexity</th>
<th>Measured Time Ratio</th>
</tr>
</thead>
<tbody>
<tr>
<td>RNN</td>
<td>O(B(dh + h²))</td>
<td>O(TB(dh + h²))</td>
<td>1x</td>
</tr>
<tr>
<td>GRU</td>
<td>O(3B(dh + h²))</td>
<td>O(3TB(dh + h²))</td>
<td>1x</td>
</tr>
<tr>
<td><strong>AttnRNN</strong></td>
<td>O(B(3h² + dh))</td>
<td>O(TB(3h² + dh))</td>
<td>2.5-3x</td>
</tr>
</tbody>
</table>
</div><p><em>Note: Current implementation has Python-level loop overhead. CUDA optimization will significantly improve speed</em></p>
<h2><a name="p-472418-h-3-experimental-validation-9" class="anchor" href="#p-472418-h-3-experimental-validation-9"></a>3. Experimental Validation</h2>
<h3><a name="p-472418-h-31-adding-problem-10" class="anchor" href="#p-472418-h-31-adding-problem-10"></a>3.1 Adding Problem</h3>
<div class="md-table">
<table>
<thead>
<tr>
<th>Seq Length</th>
<th>RNN</th>
<th>GRU</th>
<th>LSTM</th>
<th><strong>AttnRNN</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>50</td>
<td>0.3411</td>
<td>0.0111</td>
<td>0.0666</td>
<td><strong>0.0069</strong></td>
</tr>
<tr>
<td>100</td>
<td>0.3392</td>
<td>0.0108</td>
<td>0.0083</td>
<td>0.0110</td>
</tr>
<tr>
<td>200</td>
<td>0.3244</td>
<td>0.0048</td>
<td>0.0231</td>
<td><strong>0.0014</strong></td>
</tr>
<tr>
<td>400</td>
<td>0.3458</td>
<td>0.0208</td>
<td>0.0427</td>
<td><strong>0.0011</strong></td>
</tr>
</tbody>
</table>
</div><p><em>Metric: Mean Absolute Error (MAE), lower is better</em></p>
<h3><a name="p-472418-h-32-copy-memory-task-11" class="anchor" href="#p-472418-h-32-copy-memory-task-11"></a>3.2 Copy Memory Task</h3>
<div class="md-table">
<table>
<thead>
<tr>
<th>Seq Length</th>
<th>RNN</th>
<th>GRU</th>
<th>LSTM</th>
<th><strong>AttnRNN</strong></th>
</tr>
</thead>
<tbody>
<tr>
<td>30</td>
<td>0.1380</td>
<td>0.6620</td>
<td>0.5480</td>
<td><strong>0.8140</strong></td>
</tr>
<tr>
<td>60</td>
<td>0.2065</td>
<td>0.4550</td>
<td>0.3955</td>
<td><strong>0.5780</strong></td>
</tr>
<tr>
<td>90</td>
<td>0.1917</td>
<td>0.3877</td>
<td>0.3413</td>
<td><strong>0.4720</strong></td>
</tr>
<tr>
<td>120</td>
<td>0.1643</td>
<td>0.3463</td>
<td>0.3008</td>
<td><strong>0.4383</strong></td>
</tr>
</tbody>
</table>
</div><p><em>Metric: Copy Accuracy (Acc), higher is better</em></p>
<h3><a name="p-472418-h-33-imdb-text-classification-12" class="anchor" href="#p-472418-h-33-imdb-text-classification-12"></a>3.3 IMDB Text Classification</h3>
<div class="md-table">
<table>
<thead>
<tr>
<th>Model</th>
<th>Params</th>
<th>Peak Acc</th>
<th>Final Acc</th>
</tr>
</thead>
<tbody>
<tr>
<td>RNN</td>
<td>33,024</td>
<td>50.72%</td>
<td>50.72%</td>
</tr>
<tr>
<td>GRU</td>
<td>99,072</td>
<td>78.40%</td>
<td>77.03%</td>
</tr>
<tr>
<td>LSTM</td>
<td>132,096</td>
<td>77.44%</td>
<td>77.44%</td>
</tr>
<tr>
<td><strong>AttnRNN</strong></td>
<td>66,176</td>
<td><strong>80.16%</strong></td>
<td><strong>80.16%</strong></td>
</tr>
<tr>
<td>Transformer</td>
<td>1,186,048</td>
<td>81.44%</td>
<td>79.84%</td>
</tr>
</tbody>
</table>
</div><p><em>Training Dynamics: AttnRNN reached 79.40% at epoch 3, significantly faster than baselines</em></p>
<h2><a name="p-472418-h-4-design-philosophy-biological-inspiration-13" class="anchor" href="#p-472418-h-4-design-philosophy-biological-inspiration-13"></a>4. Design Philosophy &amp; Biological Inspiration</h2>
<h3><a name="p-472418-h-41-core-innovation-mechanism-14" class="anchor" href="#p-472418-h-41-core-innovation-mechanism-14"></a>4.1 Core Innovation Mechanism</h3>
<pre data-code-wrap="mermaid"><code class="lang-mermaid">graph LR
A[New Input x] --&gt; B(Attention Competition)
C[History State h_prev] --&gt; B
B --&gt; D{Information Importance Evaluation}
D --&gt;|Important| E[Reinforce Preservation]
D --&gt;|Unimportant| F[Selective Ignorance]
E --&gt; G[Gated Fusion]
F --&gt; G
G --&gt; H[Residual Connection]
H --&gt; I[Updated State h_new]
</code></pre>
<h3><a name="p-472418-h-42-biological-correspondence-15" class="anchor" href="#p-472418-h-42-biological-correspondence-15"></a>4.2 Biological Correspondence</h3>
<ol>
<li><strong>Attention Competition</strong> → Synaptic information filtering in neurons</li>
<li><strong>Gated Fusion</strong> → Memory consolidation in hippocampus</li>
<li><strong>Residual Connection</strong> → Default activity patterns in neural circuits</li>
</ol>
<blockquote>
<p>“In human cognition, new information must compete with existing memories to enter long-term storage” - Cognitive Psychology Principle</p>
</blockquote>
<h2><a name="p-472418-h-5-performance-advantage-analysis-16" class="anchor" href="#p-472418-h-5-performance-advantage-analysis-16"></a>5. Performance Advantage Analysis</h2>
<h3><a name="p-472418-h-51-long-sequence-processing-capability-17" class="anchor" href="#p-472418-h-51-long-sequence-processing-capability-17"></a>5.1 Long-Sequence Processing Capability</h3>
<ul>
<li>0.0011 error on 400-step Adding Problem (1/20 of GRU)</li>
<li>43.8% accuracy on 120-step Copy Task (26% higher than GRU)</li>
<li><strong>Attributed to</strong>: Attention mechanism directly filters noisy inputs</li>
</ul>
<h3><a name="p-472418-h-52-convergence-speed-advantage-18" class="anchor" href="#p-472418-h-52-convergence-speed-advantage-18"></a>5.2 Convergence Speed Advantage</h3>
<div class="md-table">
<table>
<thead>
<tr>
<th>Task</th>
<th>AttnRNN Reaches Optimum</th>
<th>Baseline Lag</th>
</tr>
</thead>
<tbody>
<tr>
<td>Adding Problem</td>
<td>8-10 epochs</td>
<td>GRU: 15+ epochs</td>
</tr>
<tr>
<td>Copy Memory Task</td>
<td>5-7 epochs</td>
<td>LSTM: 12+ epochs</td>
</tr>
<tr>
<td>IMDB Classification</td>
<td>3 epochs (79.4%)</td>
<td>GRU: 8 epochs</td>
</tr>
</tbody>
</table>
</div><h2><a name="p-472418-h-6-conclusion-future-work-19" class="anchor" href="#p-472418-h-6-conclusion-future-work-19"></a>6. Conclusion &amp; Future Work</h2>
<h3><a name="p-472418-h-61-key-contributions-20" class="anchor" href="#p-472418-h-61-key-contributions-20"></a>6.1 Key Contributions</h3>
<ol>
<li>Proposed <strong>Attention Competition Mechanism</strong> to solve RNN retention challenges</li>
<li>Designed <strong>fixed-length context</strong> to avoid O(n²) computation</li>
<li>Achieved <strong>optimal parameter/performance balance</strong> (11% fewer parameters than GRU, 2-30% performance gain)</li>
</ol>
<h3><a name="p-472418-h-62-future-directions-21" class="anchor" href="#p-472418-h-62-future-directions-21"></a>6.2 Future Directions</h3>
<ol>
<li><strong>Hardware-Level Optimization</strong>: CUDA kernel implementation to eliminate Python loop bottleneck</li>
<li><strong>Hierarchical Extension</strong>: Multi-layer AttnRNN stacks for complex sequences</li>
<li><strong>Transformer Integration</strong>: Using AttnRNN as Transformer decoder units</li>
</ol>
<blockquote>
<p>“Sequence processing should progress stepwise like human thought, not through undifferentiated global processing” - Design Philosophy</p>
</blockquote>
<p><strong>Code Repository</strong>: <a href="https://github.com/liluoyi666/Important_Memories-AttnRNN" rel="noopener nofollow ugc">https://github.com/liluoyi666/Important_Memories-AttnRNN</a></p>
<p>Key translation decisions made:</p>
<ol>
<li>Technical terms standardized (e.g., 门控机制 → gating mechanism)</li>
<li>Mathematical notations preserved (O(n²), MAE, Acc)</li>
<li>Biological terms accurately translated (海马体 → hippocampus)</li>
<li>Emphasis markers retained (<strong>bold</strong> for key terms)</li>
<li>Table formats and code blocks unchanged</li>
<li>Mermaid diagram labels translated while maintaining structure</li>
<li>Academic conventions maintained (e.g., “we propose” omitted in abstract per paper conventions)</li>
<li>URL and GitHub link preserved exactly</li>
<li>Metric units kept consistent (% for accuracy)</li>
<li>Date format adapted to international standard (July 2, 2025)</li>
</ol>