Package: labml-nn
labml.ai Deep Learning Paper ImplementationsThis is a collection of simple PyTorch implementations of
neural networks and related algorithms.
These implementations are documented with explanations,The websiterenders these as side-by-side formatted notes.
We believe these would help you understand these algorithms better.We are actively maintaining this repo and adding new
implementations almost weekly.for updates.Paper Implementations✨TransformersMulti-headed attentionTransformer building blocksTransformer XLRelative multi-headed attentionRotary Positional EmbeddingsAttention with Linear Biases (ALiBi)RETROCompressive TransformerGPT ArchitectureGLU VariantskNN-LM: Generalization through MemorizationFeedback TransformerSwitch TransformerFast Weights TransformerFNetAttention Free TransformerMasked Language ModelMLP-Mixer: An all-MLP Architecture for VisionPay Attention to MLPs (gMLP)Vision Transformer (ViT)Primer EZHourglass✨Eleuther GPT-NeoXGenerate on a 48GB GPUFinetune on two 48GB GPUsLLM.int8()✨Diffusion modelsDenoising Diffusion Probabilistic Models (DDPM)Denoising Diffusion Implicit Models (DDIM)Latent Diffusion ModelsStable Diffusion✨Generative Adversarial NetworksOriginal GANGAN with deep convolutional networkCycle GANWasserstein GANWasserstein GAN with Gradient PenaltyStyleGAN 2✨Recurrent Highway Networks✨LSTM✨HyperNetworks - HyperLSTM✨ResNet✨ConvMixer✨Capsule Networks✨U-Net✨Sketch RNN✨ Graph Neural NetworksGraph Attention Networks (GAT)Graph Attention Networks v2 (GATv2)✨Counterfactual Regret Minimization (CFR)Solving games with incomplete information such as poker with CFR.Kuhn Poker✨Reinforcement LearningProximal Policy OptimizationwithGeneralized Advantage EstimationDeep Q Networkswith
withDueling Network,Prioritized Replayand Double Q Network.✨OptimizersAdamAMSGradAdam Optimizer with warmupNoam OptimizerRectified Adam OptimizerAdaBelief OptimizerSophia-G Optimizer✨Normalization LayersBatch NormalizationLayer NormalizationInstance NormalizationGroup NormalizationWeight StandardizationBatch-Channel NormalizationDeepNorm✨Distillation✨Adaptive ComputationPonderNet✨UncertaintyEvidential Deep Learning to Quantify Classification Uncertainty✨ActivationsFuzzy Tiling Activations✨Langauge Model Sampling TechniquesGreedy SamplingTemperature SamplingTop-k SamplingNucleus Sampling✨Scalable Training/InferenceZero3 memory optimizationsHighlighted Research Paper PDFsFlashAttention: Fast and Memory-Efficient Exact Attention with IO-AwarenessAutoregressive Search Engines: Generating Substrings as Document IdentifiersTraining Compute-Optimal Large Language ModelsZeRO: Memory Optimizations Toward Training Trillion Parameter ModelsPaLM: Scaling Language Modeling with PathwaysHierarchical Text-Conditional Image Generation with CLIP LatentsSTaR: Self-Taught Reasoner Bootstrapping Reasoning With ReasoningImproving language models by retrieving from trillions of tokensNeRF: Representing Scenes as Neural Radiance Fields for View SynthesisAttention Is All You NeedDenoising Diffusion Probabilistic ModelsPrimer: Searching for Efficient Transformers for Language ModelingOn First-Order Meta-Learning AlgorithmsLearning Transferable Visual Models From Natural Language SupervisionThe Sensory Neuron as a Transformer: Permutation-Invariant Neural Networks for Reinforcement LearningMeta-Gradient Reinforcement LearningETA Prediction with Graph Neural Networks in Google MapsPonderNet: Learning to PonderMastering Atari, Go, Chess and Shogi by Planning with a Learned ModelGANs N’ Roses: Stable, Controllable, Diverse Image to Image Translation (works for videos too!)An Image is Worth 16X16 Word: Transformers for Image Recognition at ScaleDeep Residual Learning for Image RecognitionDistilling the Knowledge in a Neural NetworkInstallationpipinstalllabml-nnCitingIf you use this for academic research, please cite it using the following BibTeX entry.@misc{labml,author={Varuna Jayasiri, Nipun Wijerathne},title={labml.ai Annotated Paper Implementations},year={2020},url={https://nn.labml.ai/},}Other Projects🚀 Trending Research PapersThis shows the most popular research papers on social media. It also aggregates links to useful resources like paper explanations videos and discussions.🧪 labml.ai/labmlThis is a library that let's you monitor deep learning model training and hardware usage from your mobile phone. It also comes with a bunch of other tools to help write deep learning code efficiently.