Package: dynastes
DynastesDescriptionCollection of various of my custom TensorFlow-Keras 2.0+ layers, utils and such.The focus of this library is on time-series, audio, DSP and GAN related networksInstallpipinstalldynastesLayersGeneralized Multihead-AttentionWorks with any 1D-transform for QKVWorks with any 1D-attention layerTensor2Tensor 1D-Attention:Dot Product Attention*Dot Product Relative Self-Attention (Masked/Unmasked)*Sparse Attention (Truncated)Local Attention (Masked/Unmasked)*Tensor2Tensor 2D-Attention:Dot Product Relative Self-Attention (Unmasked)*Localized Attention (1D and 2D)*Perform attention within "kernels", a bit like convolutionTime-Delay Neural Network LayersAdaptive Normalization LayersAdaptive Instance Normalization (StyleGAN etc)Adaptive Layer-Instance Normalization (U-GAT-IT)Adaptive Group Normalization (?)*Supports multiquery attentionAll layers support Spectral Normalization of kernels:kernel_normalizer='spectral'even relative attention layers:key_embedding_normalizer='spectral'
value_embedding_normalizer='spectral'All you need to do in a GAN training is then to call network(x/z, training=True) when training generator or discriminator, updates are automatically performed on the u-variable if training=True. This is enabled by having a "normalizers" dictionary for every weight.
If you implement a custom layer that inherits from DynastesBaseLayer you can assign spectral normalization simply by passing wname_normalizer to the creation args, where wname is the name you give your weight.
This has some caveats, if you call super.get_weight(name) you get the normalized weight, not the actual var / rvarRegularizers, Normalization, Constraints, InitializersOrthogonal RegularizationSpectral NormalizationFunctions / opsnD-AttentionSupports "Multi-Query Attention" fromFast Transformer Decoding: One Write-Head is All You NeedSpectral ops:Wave <-> STFTSTFT <-> IF-Mel-spectrogramModified from GANSynth, now supports multi-channel audio and lower error using 128-bit complex representationModels / Architecture ScaffoldingProgressive Growing of GANs / StyleGAN scaffoldingEasily implement any kind of growing GAN in tf.keras by inhereting from these models:SimpleGrowingGanClassiferSimpleGrowingGanGeneratorRoadmap:More attention variants (1D, 2D,Relative,Local, Area) from T2TReimplementations of standard TF-Keras layers to support spectral normalization, etcGAN-scaffoldings (ProGAN,StyleGAN, BiGAN, BiStyleGAN?)Graph-attention/convolutional embedding layersupply CSR-matrix/ces for node-connectionsif training=True then it updates, otherwise reads stored valueWhy?Keras in TensorFlow 2.0 is nice, but sometimes you need exotic layers and functions that are cumbersome to implement, and I've found myself reimplementing or porting parts of T2T and other things for work and in private, over and over. This library aims to consolidate some of that and maintain tests for it.The name "Dynastes"Dynastes is a genus of large beetles belonging to the subfamily Dynastinae, rhinoceros [ῥῑνόκερως (rhīnókerōs)] beetles and it is also the name of the son of Heracles and Erato (Thespius 49th daughter). This is a play on the word Keras [κέρας (kéras, “horn”)].Notes about sources:This repository borrows code heavily from:Tensor2TensorTensorFlow 2.0 TutorialsCode is copied for stability onto this repository and attribution available when possible