Package: gaussian-adaptive-attention
Multi-Head Gaussian Adaptive AttentionGaussianAdaptiveAttention is a PyTorch library providing modules for applying Gaussian adaptive attention mechanisms that can approximate any Probability Distribution to derive attention weights. This library offers a novel approach to enhance neural network models with adaptive attention capabilities, inspired by Gaussian distribution principles.FeaturesCustomizable Gaussian Attention: Tailor the attention mechanism to suit various neural network architectures.Multiple Attention Heads: Support for multiple attention heads for complex tasks.PyTorch Integration: Seamlessly integrates with existing PyTorch models.InstallationInstall GaussianAdaptiveAttention easily using pip:pip3installgaussian-adaptive-attentionORpip3installgit+https://github.com/gioannides/Gaussian-Adaptive-Attention.gitRequirementsPython 3.xPyTorch (latest version recommended)UsageImport the GaussianAdaptiveAttention and MultiHeadGaussianAdaptiveAttention modules and integrate them into your PyTorch models:importtorchfromgaussian_adaptive_attentionimportGaussianAdaptiveAttention,MultiHeadGaussianAdaptiveAttention,GaussianBlockExample UsageThis example demonstrates the use of theGaussianBlockclass, which encapsulates multiple layers of Gaussian Adaptive Attention.importtorchimporttorch.nnasnnfromgaussian_adaptive_attentionimportGaussianBlocknorm_axes=[1,1,1]# Axes for each layer in the GaussianBlock.num_heads=[4,4,4]# Number of attention heads for each layer.num_gaussians=[5,5,5]# Number of Gaussians per head for each layer.num_layers=3# Total number of layers in the GaussianBlock.padding_value=None# Padding value for sequences in the input tensor.eps=1e-8# Small epsilon value for numerical stability.# Initialize the GaussianBlockattention_block=GaussianBlock(norm_axes,num_heads,num_gaussians,num_layers,padding_value,eps)# Example neural network with GaussianBlockclassExampleNetwork(nn.Module):def__init__(self,input_dim,output_dim):super(ExampleNetwork,self).__init__()# Initialize GaussianBlock for attention mechanismself.attention_block=attention_block# Initialize a linear layerself.linear=nn.Linear(input_dim,output_dim)defforward(self,x):# Apply GaussianBlock for attentionx=self.attention_block(x)# Apply linear layerx=self.linear(x)returnx# Example usageinput_dim=128output_dim=128model=ExampleNetwork(input_dim,output_dim)input_tensor=torch.rand(10,input_dim)# Example input tensoroutput=model(input_tensor)PaperIf you use the Gaussian Adaptive Attention, please cite our paper (https://arxiv.org/abs/2401.11143). The source code for the experiments in the paper is coming soon!@misc{ioannides2024gaussian,
      title={Gaussian Adaptive Attention is All You Need: Robust Contextual Representations Across Multiple Modalities}, 
      author={Georgios Ioannides and Aman Chadha and Aaron Elkins},
      year={2024},
      eprint={2401.11143},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}ContributingContributions to the GaussianAdaptiveAttention library are welcome!Report Issues: Submit an issue on GitHub if you find bugs or potential improvements.Submit Pull Requests: Feel free to fork the repository and submit pull requests with your enhancements.LicenseThis project is licensed under Apache-2.0 - see the LICENSE file in the repository for details.