Package: sig-networks
SigNetworks: Sequential Path Signature NetworksSigNetworks (sig-networks) is a PyTorch package for training and evaluating
neural networks for longitudinal NLP classification tasks.sig-networksis a
library that applies models first developed inSequential Path Signature Networks for Personalised Longitudinal Language Modelingby Tseriotou et al. (2023) which presented a novel extension of neural
sequential models using the notion of path signatures from rough path theory.InstallationSigNetworks is available on PyPI and can be installed with pip:pipinstallsig_networksNote that currentlysig_networksonly supports Python 3.8 since it relies on
thesignatorylibrary. However,
it is possible to installsignatorywith more recent Python and PyTorch
versions if you install it from source. See the installation guide in thesignatory documentationfor more details.Signatory/TorchSigNetworks depends on thepatrick-kidger/signatorylibrary for differentiable computation of path signatures/log-signatures in
PyTorch. Please see thesignatory documentationfor
installation instructions of the signatory library.A commonsignatoryinstallation issue is that the installation requires that
you already have PyTorch installed. In this case, you can try the following:# install PyTorchpipinstalltorch==1.9.0# install signatorypipinstallsignatory==1.2.6.1.9.0# install sig_networkspipinstallsig_networksIf you encounter any issues with the installation ofsignatory, please see the
FAQs in thesignatory documentation.Repo structureThe key parts of the libary are found insrc/:src/sig_networks/contains the source code for the
models and includes PyTorch modules for the various components of the models
(see below for more usage details)src/scripts/contains some helper scripts for training and
evaluating the modelsUsageThe key components in thesignature-windowmodels presented in (seeSequential Path Signature Networks for Personalised Longitudinal Language Modelingfor full details) are written as PyTorch modules which can be used in a modular
fashion. The key components are:The Signature Window Network Units (SWNUs):sig_networks.SWNUThere also exists the SWLSTM module which does not include the
1D-convolution layer at the start of the SWNU:sig_networks.SWLSTMThe Signature Window (Multihead-)Attention Units (SWMHAUs):sig_networks.SWMHAUAs with the SWNU, there also exists the SWMHA module which does not include
the 1D-convolution layer at the start of the SWMHAU:sig_networks.SWMHAThe SWNU-Network model:sig_networks.SWNUNetworkThe SWMHAU-Network model:sig_networks.SWMHAUNetworkThe SeqSigNet model:sig_networks.SeqSigNetThe SeqSigNet-Attention-Encoder model:sig_networks.SeqSigNetAttentionEncoderThe SeqSigNet-Attention-BiLSTM model:sig_networks.SeqSigNetAttentionBiLSTMnlpsig: Preparing the dataThe functionality to prepare the data for the SW-models and constructing paths
and inputs are found in thenlpsiglibrary which can be easily installed usingpipand comes as a dependency insig-networks.Paths can be constructed using thenlpsig.PrepareDataclass. Furthermore, there is functionality within thenlpsig.TextEncoderandnlpsig.SentenceEncoderclasses to obtain embeddings using transformers to be used as the channels in
the paths. Since we want to take path signatures within the SW-models, we need
to ensure that the number of channels in the path are low enough that we can
take the path signatures efficiently. To enable this, there are also a number of
dimensionality reduction methods in thenlpsiglibrary - seenlpsig.DimReduce.For full details, see thenlpsigGitHub repoand there are
examples of using the library in theexamples/directory.Note that for obtaining inputs to the SWNU-/SWMHA-Networks and the SeqSigNet
family models, there are helper functions in the scripts (see e.g.obtain_SWNUNetwork_inputinsrc/scripts/swnu_network_functions.pyandobtain_SeqSigNet_inputinsrc/scripts/seqsignet_functions.py).
There is also examples run-throughs in theexamples/directory:Training a SWNU-Network model for Anno-MI client-talk-type predictionTraining a SeqSigNet model for Anno-MI client-talk-type predictionUsing the SWNU and SWMHAU modulesThe Signature Window units (SWNU and SWMHAU) accept a batch of streams and
returns a batch of feature representations. For example:fromsig_networks.swnuimportSWNUimporttorch# initialise a SWNU objectswnu=SWNU(input_channels=10,hidden_dim=5,log_signature=False,sig_depth=3,pooling="signature",BiLSTM=True,)# create a three-dimensional tensor of batched streams# shape [batch, length, channels] where channels = 10streams=torch.randn(2,20,10)# pass the streams through the SWNUfeatures=swnu(streams)# features is a two-dimensional tensor of shape [batch, signature_channels]features.shapeThe SWMHAU is similar to the SWNU, but rather than having an LSTM to process the
signature streams, we have a multihead-attention layer. For example:fromsig_networks.swmhauimportSWMHAUimporttorch# initialise a SWMHAU objectswmhau=SWMHAU(input_channels=10,output_channels=5,log_signature=False,sig_depth=3,num_heads=5,num_layers=1,dropout_rate=0.1,pooling="signature",)# create a three-dimensional tensor of batched streams# shape [batch, length, channels] where channels = 10streams=torch.randn(2,20,10)# pass the streams through the SWMHAUfeatures=swmhau(streams)# features is a two-dimensional tensor of shape [batch, signature_channels]features.shapeNote in the above, we used thepooling="signature"option. This means that at
the end of the SWNU/SWMHAU, we will take a final signature of the streams to get
a fixed-length feature representation for each item in the batch. There are
other options such as taking the final LSTM hidden state for SWNU (setpooling="lstm"), or using a CLS pooling for SWMHAU (setpooling="cls").
There is another option wherepooling=Nonewhich means that the SWNU/SWMHAU
where no pooling is applied at the end of the SWNU/SWMHAU and the output is a
three-dimensional tensor of shape[batch, length, hidden_dim].Using the network modelsThe library also has the SWNU-Network and SeqSigNet models as introduced inSequential Path Signature Networks for Personalised Longitudinal Language Modeling.Since then, there have been developments of other models which utilise the SWNUs
and SWMHAUs discussed above. Each of these models are avaliable as PyTorch
modules which can be initialised and trained in the usual way.For SWNU-Network and SWMHAU-Network models, they expect two inputs:path: a batch of streams of shape[batch, length, channels]- these get
processed by the SWNU/SWMHAUfeatures: a batch of features of shape[batch, features]- these get
concatenated with the output of the SWNU/SWMHAU to be fed into a FFN layerFor SeqSigNet models (e.g. SeqSigNet, SeqSigNet-Attention-Encoder,
SeqSigNet-Attention-BiLSTM), they also expect two inputs but the path is
slightly different:path: a batch of streams of shape[batch, units, length, channels]- each
of the units for each batch will get processed by the SWNU/SWMHAU.
Afterwards, there is a global network to process the outputs of the
SWNU/SWMHAU in order to pool the outputs into a single fixed-length feature
represenation for the history. The global network can either be a BiLSTM (in
the case of SeqSigNet and SeqSigNet-Attention-BiLSTM) or a Transformer
Encoder (in the case of SeqSigNet-Attention-Encoder).features: a batch of features of shape[batch, features]- these get
concatenated with the output of the global network (either BiLSTM or a
Transformer Encoder) that processes the outputs of SWNU and SWMHAU to be fed
into a FFN layerExample experimentsIn theexamples/directory, there are some example experiments
using the library to comapre the SW-models with other baseline models such as a
simple FFN, a BiLSTM model on the sentence-transformer representations and using
a pre-trained Transformer model for classification.Pre-commit and lintersTo take advantage ofpre-commit, which will automatically format your code and
run some basic checks before you commit:pip install pre-commit  # or brew install pre-commit on macOS
pre-commit install  # will install a pre-commit hook into the git repoAfter doing this, each time you commit, some linters will be applied to format
the codebase. You can also/alternatively runpre-commit run --all-filesto run
the checks.