Package: cntkx
CNTKxCNTKx is a deep learning library that builds on and extends Microsoft Cognitive ToolkitCNTK.
Despite the last planned release of cntk 2.7, cntkx will continue to be in active development, more models and pre-built components coming soon!Feel free to open an issue for any request or a PR to contribute :)Installationcntkx is written in pure python and cntk is a dependency to it. Please get a working installation of cntk first. Then:pip install cntkxcntkx only works withpython>=3.6Available ComponentsopsDescriptionfloor_divisionelement-wise floor_divisionremainderelement-wise remainder of divisionscalarcast tensor to scalar (1,)cumsumCumulative summation along axisupsampleUpsample by k factor (for image)centre_cropCrop centre of imageswishActivationmishActivationhardmaxActivationerfError functiongeluGaussian Error Linear Unit functiongelu_fastfast approximation of Gaussian Error Linear Unit functionsequence.padPad at start or end of sequence axissequence.pad_toPad a sequence to have the same length as another sequencesequence.lengthlength of sequencesequence.positionposition of every sequence elementsequence.stridestrides across sequential axissequence.joinjoins two sequence along their sequential axissequence.windowcreates sliding window along the sequence axissequence.window_causalcreates causal sliding window along the sequence axissequence.reversereverses the items along the dynamic sequence axissequence.reduce_meancalculates the mean along the dynamic sequence axissequence.pad_ctc_labelspadded ctc labels to be the same sequence length as the network outputsequence.reduce_concat_pooldrop-in replace for sequence.lastrandom.sampleSamples an unnormalised log probability distributionrandom.sample_with_biasSamples an unnormalised log probability distribution over-weighted to more probable classesrandom.sample_top_kSamples from the top_k of an unnormalised log probability distributionbatchmatmulBatch Matrix Multiplication on a static batch axis, similar to tf.matmulLayersDescriptionQRNNQuasi-Recurrent Neural NetworkRecurrenceWith option to applyVariationalDroppoutPyramidalBiRecurrencePyramidal bi-directional recurrenceVariationalDropoutSingle binary dropout mask for entire sequenceSinusoidalPositionalEmbeddingNon-learnable positional embedding (no max sequence length)PositionalEmbeddingLearnable Positional Embedding (used in BERT)BertEmbeddingsBERT Embeddings (word + token_type + positional)BertPoolerPooler used in BERTSpatialPyramidPoolingFixed pooled representation regardless of image input sizeGatedLinearUnitGated Convolutional Neural NetworkScaledDotProductAttentionAttention used in BERT and Transformer (aka 'attention is all you need')MultiHeadAttentionAttention used in BERT and Transformer (aka 'attention is all you need')GaussianWindowAttentionWindowed attention instead of conventional attention where everything is attended at the same timeSequentialDenseApplies Dense to a window of sequence item along sequence axisSequentialMaxPoolingMax pool across sequential axis and static axesSequentialAveragePoolingAverage pool across sequential axis and static axesSequentialConcatPoolingConcat Average and Mean pool across sequential axis and static axesvFSMNVectorised Feedforward Sequential Memory NetworkscFSMNCompact Feedforward Sequential Memory NetworksBiRecurrenceBiRecurrence recurrent layer with weight tying option to half parameter requirementGlobalConcatPoolingGlobal spatial concat pooling of ave and meanFilterResponseNormalizationDrop in replacement for batch norm with superior performanceBoomMore parametrically efficient alternative to Position-Wise FeedForward layer found in TransformerGaussianAttentionSeqImageMemory efficient attention that used 2d gaussian filters for imagesSequenceDropoutDropout entire sequence elementsSEBlockSqueeze and Excitation blockSequenceSEBlockSqueeze and Excitation block for variable width image sequenceSIRENSinusoidal Representation NetworkLinearAttentionLinearised form of dot-product attention with linear memory complexityLinearAttentionModelWrapper toLinearAttentionBlocksDescriptionWeightDroppedLSTMA form of regularised LSTMIndyLSTMA parameter efficient form of LSTMIndRNNa RNN with long memory and can be stacked deeplyLossDescriptiongaussian_mdn_lossloss function when using Mixture density networkfocal_loss_with_softmaxA kind of cross entropy that handles extreme class imbalancecross_entropy_with_softmaxAddedlabel smoothing regularisationin cross entropy with softmaxgeneralised_robust_barron_lossgeneralised robust lossModelsDescriptionVGGImage ClassificationUNETSemantic SegmentationTransformerLanguage ModellingMDNMixture Density NetworksPre-trained modelsDescriptionBertBidirectional Encoder Representations from Transformersfwd_wt103.hdf5The weight parameters of the fastai's pytorch model. To be used to initialisePretrainedWikitext103LanguageModelfwd_wt103.cntkThe converted cntk model of fastai's pytorch model. To be used withC.load_modelfwd_wt103.onnxThe converted ONNX model of fastai's pytorch model.LearnersDescriptionCyclicalLearningRatea method to eliminate the need to find best value and schedule for learning rateRAdama variant ofAdamthat doesn't require any warmupMiscDescriptionCTCEncoderHelper class to convert data into a format acceptable for cntk's ctc implementationC# CNTK TutorialsThis library is implemented in pure cntk python API. For help in cntk c#, you can refer to the two repositorydeep-learning-with-csharp-and-cntkandDeepBelief_Course4_Examples.F# CNTKFor the F# wrapper of CNTK please visitFsCNTK,
it also contains some example implementations like seq2seq, autoencoder, LSTM, GAN.News2021-04-04Added new kernel method intoLinearAttentionPreviousLinearAttentionuses implemntation fromTransformers are RNNs:Fast Autoregressive Transformers with Linear Attentionthat useselu.eluas kernel may not perform well in certain task (e.g. ASR) as documented in the paper.Hence, new kernel trick usessoftmaxfromEfficient Attention: Attention with Linear Complexities.2020-09-06Addedsequence.pad_ctc_labelsPads the ctc label sequence to the same sequence length as the network output.
This should be used when the final sequence length of the network output cannot be determined
beforehand during the pre-processing of the ctc_labels. Thus, the padding is done during training runtime
instead of during the data pipeline processing.The padding token would be the last sequence element ofctc_labels.ctc_labelsshould be
a one hot encoded vector sequence. The padding token will have the value of 1 in its one-hot encoded vector.2020-06-20AddedLinearAttentionandLinearAttentionModelAdded cntk implementation ofLinearAttentionandLinearAttentionModel.
Standard dot-product attention found inTransformeris quadratic in time and gpu memory complexity
with respect to sequence length. TheC.layers.AttentionModelis also quadratic.However,LinearAttentionis a linearised form of dot-product attention. It is linear in time and gpu memory complexity
with respect to sequence length. This is especially significant sincecntkdoesn't have any checkpointing function
compared to other frameworks liketensorflowandpytorch, where it saves gpu memory at the expense of some
computation overhead. Now, withLinearAttentiontraining ofTransformercan be done oncntksince the
gpu memory requirement is drastically reduced.LinearAttentionModelis wrapper forLinearAttentionin the style ofC.layers.AttentionModel.For more details refer toTransformers are RNNs:Fast Autoregressive Transformers with Linear Attentionby Katharopoulos et al.Example:a = C.sequence.input_variable(24)
b = LinearAttention(hidden_dim=32, model_dim=24)(a, a, a)

assert b.shape == (32, )2020-06-20Addedsequence.pad_tosequence.pad_topads a shorter sequence to the same sequence length as another longer sequence.
This is especially useful in CTC training where both sequences need to have the same sequence lengthExample:ax1 = C.Axis.new_unique_dynamic_axis('ax1')
ax2 = C.Axis.new_unique_dynamic_axis('ax2')
a = C.sequence.input_variable(3, sequence_axis=ax1)
b = C.sequence.input_variable(6, sequence_axis=ax2)

c = Cx.sequence.pad_to(a, b)  # pad to same sequence length

ctc_token = C.Constant(np.array([0, 0, 1]))
d = C.element_select(C.equal(c, 0), ctc_token, c)  # swap padded zeros with ctc token2020-06-19AddedSIRENSIRENleverages periodic activation functions for implicit neural representations and demonstrate
that these networks are ideally suited for representing complex natural signals and their derivatives.The results are incredible. It is highly recommended that you check out theirproject page.More details can be found in their paperImplicit Neural Representations with PeriodicActivation Functions2020-05-01AddedSEBlockandSequenceSEBlockSEBlockandSequenceSEBlockare squeeze-excitation blocks that adaptively recalibrates channel-wise feature
responses by explicitly modelling interdependencies between channels. It was show that these blocks can be
stacked together to form SENet architectures that generalise extremely effectively across different datasets.Its the winner for ILSVRC2017 classification submission. More details can be found in the
paperSqueeze-and-Excitation Networksby Jie hu et al.2020-04-20AddedSequenceDropoutSequenceDropoutdropouts entire sequence elements along the dynamic sequence axis.2020-04-15AddedGaussianAttentionSeqImageGaussianAttentionSeqImageis a 2d spatial gaussian attention.
To use, the encoded image should be formulated as a cntk sequence (example below). This can be useful when
you are constraint by gpu memory as 2d gaussian attention is more memory efficient than standard attention.This is from the deepmind paper, DRAW: A Recurrent Neural Network for Image Generation by Gregor et al
More details can be found in the followinghttps://arxiv.org/abs/1502.04623Example:n = 5
num_channels = 3
image_height = 64
expected_image_width = 1000
image_seq = C.sequence.input_variable((num_channels, image_height))  # rgb image with variable width and fixed height
decoder_hidden_state = ...  # from decoder somewhere in the network
attended_image = Cx.layers.GaussianAttentionSeqImage(n, image_height, expected_image_width)(image_seq, decoder_hidden_state)

assert attended_image.shape == (num_channels, n, n)2020-03-29.Addedgeneralised_robust_barron_lossandsample_with_biasgeneralised_robust_barron_lossis a generalisation for generalization of the Cauchy/Lorentzian,
Geman-McClure, Welsch/Leclerc, generalized Charbonnier, Charbonnier/pseudo-Huber/L1-L2, and
L2 loss functions.Can be used as a drop-in replacement in any regression task that you have.For more details, please refer toA General and Adaptive Robust Loss Function, Jonathan T. Barron,
or thisvideo.
It is the Best Paper Award Finalist in CVPR 2019.Implementedsample_with_biasto sample more likely classes as a replacement
forsample_top_kwhich cannot be used inside aUnfoldFrom.sample_with_biasreduces sampling variance especially when you have a long tail.2020-02-11.AddedBoomBoom layer from SHA-RNN by S. Merity creator of QRNN. Alternative to PositionwiseFeedForward. Serves the same function as
PositionwiseFeedForward found in transformer.Boom layer minimizes computation and removes an entire matrix of parameters compared to traditional down-projection layers.For more detail please read:Single Headed Attention RNN: Stop Thinking With Your Headby Stephen Merity.2019-12-03.AddedFilterResponseNormalizationandThresholdedLinearUnitAdded cntk implementation ofFilterResponseNormalization. Filter Response Normalization (FRN) layer
is a novel combination of a normalization and an activation function,
that can be used as a drop-in replacement for other normalizations and activations.The method operates on each activation map of each batch sample
independently, eliminating the dependency on other batch samples or channels of the same sample.
The method outperforms BN and all alternatives in a variety of settings for all batch sizes.
FRN layer performs â‰ˆ0.7âˆ’1.0% better on top-1 validation accuracy than BN with large mini-batch sizes on
Imagenet classification on InceptionV3 and ResnetV2-50 architectures. Further, it performs >1% better
than GN on the same problem in the small mini-batch size regime. For object detection problem on COCO dataset,
FRN layer outperforms all other methods by at least 0.3âˆ’0.5% in all batch size regimes.Please refer to the paperFilter Response Normalization Layer: Eliminating Batch Dependence
in the Training of Deep Neural Networksfor more details .2019-11-04.Addedops.floor_division,ops.remainder,sequence.window_causalandSequentialDenseAdd in new operations stated above.sequence.windownow has an additional argument that lets you control striding.sequence.window_causalcreates causal window that doesn't leak future information into the past (preserve causality).SequentialDenseconvenience layer added to apply dense to a window of sequence item,
much likeSequentialConvolutionbut with better memory performance.2019-11-04.AddedSequentialConcatPooling,Cx.Sequence.reduce_concat_poolandGlobalConcatPoolingCx.Sequence.reduce_concat_poolconcatenates the last item in the sequence axis with the summarisation
of the sequence represented byreduce_maxandreduce_meanof the sequence axis. AnytimeC.sequence.lastis used,
this can be a drop-in replacement.Example:n = 32
a = C.sequence.input_variable(n)
b = Cx.sequence.reduce_concat_pool(a)

assert b.shape == (n * 3, )SequentialConcatPoolingdoes spatial concat pooling over the sequential axis.
Concat pooling is the concatenation of both average pooling and max pooling. In any situation where max or ave
pooling is appropriate, concat pooling can be used as a drop-in replacement and achieve improvements in performance.Example:a = C.sequence.input_variable((3, 10))
b = SequentialConcatPooling(filter_shape=(2, 2), strides=2)(a)

assert b.shape == (6, 10)

n = [np.random.random((3, 10)).astype(np.float32),
     np.random.random((3, 10)).astype(np.float32),
     np.random.random((3, 10)).astype(np.float32),
     np.random.random((3, 10)).astype(np.float32), ]

print(b.eval({a: n}))GlobalConcatPoolingis the standard spatial concat pooling of both max pool and ave pool.2019-10-15.Addedmishactivation function andRAdamlearner.mishis an activation function is introduced inMish: A Self Regularized Non-Monotonic Neural Activation Functionby Diganta Misra. Experiments show thatmishtends to work better than both ReLU and Swish along with other standard
activation functions in many deep networks across challenging datasets. For instance,
in Squeeze Excite Net-18 for CIFAR 100 classification, the network with Mish had an increase in
Top-1 test accuracy by 0.494% and 1.671% as compared to the same network with Swish and ReLU respectively.
The similarity to Swish along with providing a boost in performance and its simplicity in implementation
makes it easier for researchers and developers to use Mish in their Neural Network Models.This activation function is adopted infast ai.Rectified Adam orRAdamis aAdamoptimiser variant that doesn't require a warmup schedule, whichAdamtends
to need to maintain stability. In this cntk implementation, we added aRAdamlike optimiser based
on the work ofOn the adequacy of untuned warmup for adaptive optimizationby Jerry Ma and Denis Yarats.RAdamis adopted infast aitoo.2019-09-30.AddedBiRecurrencewith weight tyingMade improvement to weight tying of BiRecurrence by have one parameter tensor token for every state
in the step function per direction (forward and backward). This will allow forward and backward token
to have more representational flexibility. Previously, all states use the same forward or backward token.2019-09-06.AddedBiRecurrencewith weight tyingAdd a wrapper to create a bidirectional recurrent layer usingBiRecurrence. Included in the implementation
is an option to half the number of parameters required by  bidirectional recurrent layer. This is done
by only using one recurrent unit to do both forward and backward computation instead of the usual two.
A forward and backward token is used to initialise the hidden state so that the recurrent unit can tell
the directionality.More details can be found in the paperEfficient Bidirectional Neural Machine TranslationExample:a = C.sequence.input_variable(10)
b = BiRecurrence(LSTM(100), weight_tie=True)(a)

assert b.shape == (200, )2019-08-29.AddedPretrainedWikitext103LanguageModelCNTK implementation of Fast AI's Universal  Language  ModelFine-tuning (ULMFiT) English model has been added.
This language model was trained on Wikitext-103 and can be used as a base model for any downstream language task.It is also much more efficient to run compare to BERT and other Transformer language models.For more details, you can refer to the original paperhereExample:vocab_size = 238462
converted_hdf5_model_file_path = 'PATH/fwd_wt103.hdf5'  # this is not the original pytorch model
lm = PretrainedWikitext103LanguageModel(converted_hdf5_model_file_path)

a = C.sequence.input_variable(vocab_size)
prediction = lm(a)  # next-word-prediction
features = prediction.features  # features of tokens

assert prediction.shape == (vocab_size, )
assert features.shape == (400, )ModelDescriptionfwd_wt103.hdf5The weight parameters of the fastai's pytorch model. To be used to initialisePretrainedWikitext103LanguageModelfwd_wt103.cntkThe converted cntk model of fastai's pytorch model. To be used withC.load_modelfwd_wt103.onnxThe converted ONNX model of fastai's pytorch model.itos_wt103.pklTokens used in pretrained model2019-08-08.Addedcntkx.ops.geluandcntkx.ops.gelu_fastAdded two cntk implementation ofgeluactivation function.geluactivation is used inBERTand OpenAI'sGPTandGPT-2.Gaussian Error Linear Unit (GELU), a high-performing neuralnetwork activation function.
The GELU nonlinearity is the expected transforma-tion of a stochastic regularizer which randomly
applies the identity or zero mapto a neuronâ€™s input.  The GELU nonlinearity weights inputs by their
magnitude,rather than gates inputs by their sign as in ReLUs.For more detail please refer toGaussian Error Linear Units (GELU)by Hendrycks and Gimpel.2019-07-04.Addedcntkx.ops.sequence.reduce_meanCalculates the mean along the dynamic sequential axis in CNTK.2019-06-26.Addedcntkx.ops.sequence.reverseAllows the sequence items along the sequence axis to be reversed. This is useful when you want to create a
Bi-directional Auto-Regressive layer because using UnfoldFrom does not work with Recurrence(go_backwards=True) and
will result in a ValueError.Example:import cntk as C
import cntkx as Cx
from cntk.layers import Recurrence, UnfoldFrom, LSTM

hidden_dim = 50
start_token = C.Constant(0, shape=(hidden_dim,))
a = C.sequence.input_variable(1, name='seq1')

b = UnfoldFrom(Recurrence(LSTM(hidden_dim), go_backwards=True))(start_token, a)

n = [np.random.random((10, hidden_dim)).astype(np.float32),]

# This raise 'ValueError: It is not allowed to have multiple different stepping directions in the same loop'
b.eval({b.arguments[0]: n})The workaround would be:import cntk as C
import cntkx as Cx
from cntk.layers import Recurrence, UnfoldFrom, LSTM

hidden_dim = 50
start_token = C.Constant(0, shape=(hidden_dim,))
a = C.sequence.input_variable(1, name='seq1')
a_reversed = Cx.sequence.reverse(a)

b = UnfoldFrom(Recurrence(LSTM(hidden_dim)))(start_token, a_reversed)  # remove go_backwards=True

n = [np.random.random((10, hidden_dim)).astype(np.float32),]    
b.eval({b.arguments[0]: n})  # this will run just fine2019-06-21.Addedcntkx.ops.random.sample_top_kCNTK implementation of that allows sampling of the top_k unnormalised log-probabilities distribution.
This is useful text (or sequence) generation where it is known that greedy decoding will cause
text degeneration.For more details on this, please refer toA curious case of neural text degenerationExample:import cntk as C
import cntkx as Cx

a = C.input_variable(5)
b = Cx.random.sample_top_k(a, k=3, num_classes=5)

n = np.array([[1, 2, 3, 4, 5],] * 1000)

results = b.eval({a: n})
assert np.sum(results[:, :2]) == 0
assert np.sum(results[:, 2:]) == 10002019-05-04.Addedcntkx.layers.vFSMNandcntkx.layers.cFSMNCNTK implementation of Bidirectional vectorised Feedforward Sequential Memory Network (vFSMN)
and Compact Feedforward Sequential Memory Network (cFSMN).FSMN is a standard fully-connected feedforward neural network equipped
with some learnable memory blocks in its hidden layers. The memory blocks
use a tapped-delay line structure to encode the long context information into
a fixed-size representation as short-term memory mechanism.The authors claim that FSMNs can be learned much more reliably and faster than
RNNs or LSTMs due to the inherent non-recurrent model structure while significantly
outperforming RNNs in language and speech modeling.For more details please refer toFeedforward Sequential Memory Networks: A
New Structure to Learn Long-term DependencycFSMN is a compact version of FSMN that can result in a reduction of up
to 60% in model size and speed up the learning by more than 7 times while
still significantly outperforming the popular bi-direction LSTMs for both
frame-level cross-entropy (CE) criterion based training and MMI based sequence training.For more details please refer to "Compact Feedforward Sequential Memory Networks for
Large VocabularyContinuous Speech Recognition" by Zhang, et al.Example:import cntk as C
from cntkx.layers import vFSMN, cFSMN

# unidirectional vFSMN (only past conext used)
a = C.sequence.input_variable(10)
b = vFSMN(100, C.relu, num_past_context=3, num_future_context=0)(a)

assert b.shape == (100,)

# bidirectional vFSMN (enable both past and future context)
a = C.sequence.input_variable(10)
b = vFSMN(120, C.relu, num_past_context=3, num_future_context=3)(a)

assert b.shape == (120,)

# bidirectional cFSMN (enable both past and future context)
a = C.sequence.input_variable(100)
b = cFSMN(120, 50, C.relu, num_past_context=3, num_future_context=3)(a)

assert b.shape == (120,)2019-04-19.Addedcntkx.misc.CTCEncoderCNTK's CTC implementation requires that data be formatted in a particular way that's typically in acoustic
modeling but unusual in other applications. So class provides an easy way to convert data between
what users typically expect and what cntk demands.Example:
labels = ['a', 'b', 'c']
encoder = CTCEncoder(labels)labels_tensor = C.sequence.input_variable(len(encoder.classes_))  # number of classes = 4
input_tensor = C.sequence.input_variable(100)

labels_graph = cntk.labels_to_graph(labels_tensor)
network_out = model(input_tensor)

fb = C.forward_backward(labels_graph, network_out, blankTokenId=encoder.blankTokenId)

ground_truth = ['a', 'b', 'b', 'b', 'c']
seq_length = 10  # must be the same length as the sequence length in network_out

fb.eval({input_tensor: [...],
         labels_tensor: [encoder.transform(ground_truth, seq_length=seq_length)]})2019-04-14.AddedLabel Smoothing Regularization,seqeuence.windowandPyramidalBiRecurrenceAddedLabel Smoothing Regularizationincross_entropy_with_softmax.
Addedsequence.windowthat creates non-overlapping window along the sequence axis thereby reducing the
sequence length and increasing the dimension by the same factor.Implemented a convenience layer used in acoustic modelling known asPyramidalBiRecurrence. Used to create
pyramidal bi-directional LSTM (BLSTM) found in "Listen, attend and spell" by Chan et al. (https://arxiv.org/abs/1508.01211)
Typically used to down sample the sequence length to make memory and runtime manageable.2019-04-08.Addedcntkx.ops.sequence.joinAdded a newopcalledjoinwhere two sequence tensors can be joined along with sequence axis forming a longer sequence.2019-04-08.Addedcntkx.layers.SequentialAveragePoolingAdd average pooling layer that works with sequential axis. Current cntkAveragePoolingdoesn't pool across sequence elements.Example oncntkx.layers.SequentialAveragePooling# rgb image of height 25 and variable width
a = C.sequence.input_variable((3, 25))

# Convolute across image with (3, 3) kernel with stride (1, 1)
b = C.layers.SequentialConvolution(filter_shape=(3, 3), num_filters=16, stride=(1, 1), pad=True)(a)

assert b.shape == (16, 25)

# max pool (2,2) in height and width with stride (2,2) in height and width, no padding
c = SequentialAveragePooling(filter_shape=(2, 2), strides=(2, 2), pad=False)(b)

assert c.shape == (16, 12)2019-04-07.Addedcntkx.sequence.strideandcntkx.ops.scalarCx.sequence.strideenables striding across the sequential axis, selecting every integer items along the sequence.Cx.scalarconverts tensor into scalar of shape(1,)2019-04-06.AddedIndyLSTMandIndRNNCNTK implementation ofIndependently Recurrent Long Short-term Memory cells: IndyLSTMsby Gonnet and Deselaers, andIndependently Recurrent Neural Network (IndRNN): Building A Longer andDeeper RNNby Li, et al.BothIndyLSTMandIndRNNhave hidden-to-hidden weights that are diagonal matrix instead of the usual full matrix.
Thus neurons in each layer are independent from each other, and the cross-channel information is
obtained through stacking multiple layers.IndRNNallows for the use ofC.reluactivation thus allowing multipleIndRNNlayers to be stacked together deeply.IndyLSTMhas parameters linear to the number of nodes in the linear, as opposed to standard LSTM that is quadratic
makingIndyLSTMpotentially faster and smaller as a model.Authors of bothIndRNNandIndyLSTMhave claimed performance as good as or even better than regular LSTMs.Example:import cntk as C
from cntkx.layers import IndyLSTM, IndRNN, Recurrence

a = C.sequence.input_variable(10)
b = Recurrence(IndRNN(20))(a)
c = Recurrence(IndyLSTM(20))(a)

assert b.shape == c.shape == (20,)2019-03-24.Addedcntkx.layers.SequentialMaxPoolingAdd max pooling layer that works with sequential axis. Current cntkMaxPoolingdoesn't pool across sequence elements.Example oncntkx.layers.SequentialMaxPooling# rgb image of height 25 and variable width
a = C.sequence.input_variable((3, 25))

# Convolute across image with (3, 3) kernel with stride (1, 1)
b = C.layers.SequentialConvolution(filter_shape=(3, 3), num_filters=16, stride=(1, 1), pad=True)(a)

assert b.shape == (16, 25)

# max pool (2,2) in height and width with stride (2,2) in height and width, no padding
c = SequentialMaxPooling(filter_shape=(2, 2), strides=(2, 2), pad=False)(b)

assert c.shape == (16, 12)2019-03-18.Addedcntkx.learners.CyclicalLearningRateCyclical learning rate (CLR) is an implementation to that  practically eliminates the need to
experimentally find the best values and schedule  for  the global  learning  rates.Instead  of  monotonically decreasing the learning rate, this method lets the learningrate  cyclically  vary  between  reasonable  boundary  values. Training  withcyclical  learning  rates  instead of  fixed  values  achieves improved  classification
accuracy without a need to tune and often in fewer iterations.More details can be found inCyclical Learning Rates for Training Neural Networksby Leslie N. SmithThis CLR implementation can be used with the cntk training loop by adding onlytwo lines of code:model = C.layers.Dense(10)(C.input_variable(10))
sgd_momentum = C.momentum_sgd(model.parameters, 0.1, 0.9)
clr = CyclicalLeaningRate(sgd_momentum, minibatch_size=32)  # first line of code

for epoch in range(10):
    for batch in range(100):
        trainer.train_minibatch(...)
        clr.batch_step()  # second line of code (to be called after every training update)2019-03-12.Addedcntkx.ops.batchmatmulAdded Batch Matrix Multiplication. This implementation is similar
totensorflow.matmul.Example:a = C.sequence.input_variable((3, 4, 5))     # batch matrix
b = C.sequence.input_variable((3, 5, 6))     # batch matrix
c = Cx.batchmatmul(a, b)
assert c.shape == (3, 4, 6)                  # 3 is treated as a batch axis2019-03-10.AddedPretrainedBertEncoderandPretrainedBertModelBERT, the state-of-the-art language model is now available as a CNTK pretrained model.Currently, it is only tested to work withBERT-Base, Uncased(uncased_L-12_H-768_A-12) and can be
downloaded fromGoogle AIWhen you have downloadedBERT-Base, Uncased, there should be 5 files inside. You will need to.zipthree of those files into a tensorflow checkpoint file before you can load it intocntkx.Those three files are:bert_model.ckpt.data-00000-of-00001,bert_model.ckpt.index,bert_model.ckpt.meta.
Then rename the extension of.zipinto.ckptand you are good to go.Example belowtext_tensor = C.sequence.input_variable(30522)
token_type_tensor = C.sequence.input_variable(2)
filepath_to_tf_bert_model = "YOUR_FILE_DIRECTORY/bert_model.ckpt"

model = Cx.layers.PreTrainedBertModel(filepath_to_tf_bert_model, num_heads=12, dropout_rate=0.1)
b = model(text_tensor, token_type_tensor)

assert b.shape == (768,)For more details about BERT, you can find the original paperhere,
and some useful resourceshereandhere.Note:
It goes without saying also that to use these pre-trained models you will need to have tensorflow installed
since we are convert them from tensorflow models.2019-03-06.AddedPositionalEmbedding,BertEmbeddingsandPretrainedBertEmbeddingsCNTK implementation ofPositionalEmbedding,BertEmbeddingsand tf-to-cntkPreTrainedBertEmbeddings.
BERT is a state-of-the-art language model from Google AI, more details can be found inBERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.Google AI's pre-trained BERT tensorflow model can be downloadedhere.
Tensorflow would need to be installed in your environment if you intend to usePreTrainedBertEmbeddings, which
takes a tensorflow model and convert it cntk.Example forPositionalEmbeddinga = C.sequence.input_variable(12)
b = PositionalEmbedding(max_seq_length, hidden_dim)(a)

assert b.shape == (hidden_dim, )Example forBertEmbeddingstext_tensor = C.sequence.input_variable(100)
token_type_tensor = C.sequence.input_variable(2)
b = BertEmbeddings(max_seq_length, hidden_dim, 0.1)(text_tensor, token_type_tensor)

assert b.shape == (hidden_dim, )Example forPreTrainedBertEmbeddingstext_tensor = C.sequence.input_variable(30522)
token_type_tensor = C.sequence.input_variable(2)
filepath_to_tf_bert_model = "YOURFILEPATH"
embeddings = PreTrainedBertEmbeddings(filepath_to_tf_bert_model, 0.1, False)
b = embeddings(text_tensor, token_type_tensor)

assert b.shape == (768, )2019-03-02.AddedVariationalDropoutandWeightDroppedLSTMCNTK implementation ofVariational Dropoutfound inA Theoretically Grounded Application of Dropout in Recurrent Neural NetworksandWeight Dropped LSTMproposed in a salesforce research paperRegularizing and Optimizing LSTM Language Models.Weight Dropped LSTMis a regularised LSTM that uses DropConnect on hidden-to-hidden weights as a form of recurrent
regularisation. It also include application of variational dropout on the inputs and outputs of the recurrent units
for further regularisation.Variational Drpooutis a regularisation that uses same dropout mask at each time step
(i.e. across the dynamic sequence axis) as opposed to the naive application ofC.layers.Dropoutto a sequence
which will result in a different dropout mask for every tensor along the sequence axis.import cntkx as Cx
from cntkx.layers import Recurrence, WeightDroppedLSTM
import cntk as C

dropconnect_rate = 0.2
variationaldrop_rate = 0.1

seq = C.sequence.input_variable(56)
b = Recurrence(WeightDroppedLSTM(20, dropconnect_rate),
               variational_dropout_rate_input=variationaldrop_rate,
               variational_dropout_rate_output=variationaldrop_rate)(seq)

assert b.shape == (100, )

seq_dropped = VariationalDropout(0.1)(seq)

assert seq_dropped.shape == seq.shape2019-02-02.Added Gated Linear Unit / Gated CNNCNTK implementation of Gated Linear Unit (Gated CNN) founnd in Facebook AI Research Lab's paper:Language Modeling with Gated Convolutional Networks.
This paper applies a convolutional approach to language modelling with a novel Gated-CNN model.import cntkx as Cx
import cntk as C

seq = C.sequence.input_variable(56)
hidden = Cx.layers.GatedLinearUnit(window=2, hidden_dim=100)(seq)

assert hidden.shape == (100, )2019-01-21.AddedFocal Lossfor multi-class and binary classificationCNTK implementation ofFocal Lossenables the training of highly accurate dense object detectors in the
presence of vast numbers of easy background examples or dataset with extreme class imbalance (e.g. 1:1000).Focal Lossfocuses training on a sparse set of hard examples and prevents the vast number of easy negatives from
overwhelm-ing the model during training.For more details please refer toFocal Loss for Dense Object Detectionimport cntkx as Cx

Cx.focal_loss_with_softmax([[0, 0, 0.8, 0.2]], [[0, 0, 1, 0]]).eval()
array([[0.31306446]], dtype=float32)2019-01-18.Added Gaussian Window Attention ModelGaussian window attention model was first introduced by Alex Graves in
"Generating sequences with recurrent neural networks".It uses a mixture of gaussian windows to attend to
portions of the sequence as oppose to the widely used attention model introduced in
"Neural machine translation by jointly learning to align and translate" by Bahdanau, et al. that attends
to the entire sequence all at once.Gaussian window attention is also directional in its attention on the context sequence. When modeling
strongly ordered sequences, gaussian window attention will be a natural choice due to this inductive bias.import cntk as C
import cntkx as Cx

seq1 = C.Axis.new_unique_dynamic_axis('seq1')
seq2 = C.Axis.new_unique_dynamic_axis('seq2')

encoded = C.sequence.input_variable(30, sequence_axis=seq1)
query = C.sequence.input_variable(28, sequence_axis=seq2)

a = Cx.layers.GaussianWindowAttention(10)(encoded, query)

assert a.shape == (30, )"Generating sequences with recurrent neural networks" can be foundhere.
"Neural machine translation by jointly learning to align and translate" can be foundhere.2019-01-16.Added Spatial Pyramid Pooling layerSpatial pyramid pooling layer is a pooling layer than returns a fixed length representation regardless of the
image size/scale. It is frequently used for multi-size image training. It reported SOTA classification results using
a single full-image representation without fine-tuning. For more details on the paper
"Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition" by K. He, X. Zhang, S. Ren, J. Sun,
linkhere.import cntk as C
import cntkx as Cx

n = np.random.random((3, 3, 32, 32)).astype(np.float32)
a = C.input_variable((3, 32, 32))
b = Cx.layers.SpatialPyramidPooling((1, 2, 4))(a)

assert b.shape == (3 * (4 * 4 + 2 * 2 + 1),)  # representation not dependent on image size2019-01-15.Added Sinusoidal Positional Embedding andcntkx.ops.erfAdded sinusoidal positional embedding used inTransformer. For an accessible
explanation of transformer, you may look uphere.import cntk as C
import cntkx as Cx

a = C.sequence.input_variable(10)
b = SinusoidalPositionalEmbedding()(a)

assert b.shape == (10, )Addedcntkx.ops.erferror function.2019-01-12.Added Vision models: VGG16, VGG19 and UNETVGG is for image classification and UNET is for semantic segmentation. VGG is implemented for completeness
sake and should not be used for any serious classification task.Paper on VGG can be foundheretitled "Very Deep Convolutional Networks
for Large-Scale Image Recognition"Paper for UNET can be foundheretitled "U-Net: Convolutional Networks
for Biomedical Image Segmentation"VGG example:import cntk as C
import cntkx as Cx

a = C.input_variable((3, 64, 64))
b = Cx.layers.VGG19(100)(a)

assert b.shape == (100,)UNET example:import cntk as C
import cntkx as Cx

a = C.input_variable((3, 512, 512))
b = Cx.layers.UNET(num_classes=10, base_num_filters=64, pad=True)(a)

assert b.shape == (10, 512, 512)Convenience functions such ascntkx.ops.upsampleandcentre_crophave also been added.cntkx.ops.upsampleupsamples an image twice on each spatial dim.centre_cropcrops a smaller image from
a bigger one in the centre given a reference smaller image.Added Transformer attention model and associated componentsThe Transformer was first introduced in thepaper'Attention is all you need'.
The architecture is based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.
More recently,BERTwhich broke almost all SOTA language task is also based on
transformer and self-attention.import cntk as C
import cntkx as Cx

a = C.sequence.input_variable(512)
b = C.sequence.input_variable(512)

transformer = Cx.layers.Transformer()  # using default settings
decoded = transformer(a, b)

assert decoded.shape == (512, )2018-12-08.Added QRNN: Quasi-Recurrent Neural Network (QRNN) andcntkx.ops.cumsumThe QRNN provides similar accuracy to the LSTM but can be betwen 2 and 17 times faster than the
highly optimized NVIDIA cuDNN LSTM implementation depending on the use case.More details please refer to the original paperhere.import cntk as C
import cntkx as Cx

input_seq = C.sequence.input_variable(input_dim)
prediction_seq = Cx.layers.QRNN(hidden_dim=50)(input_seq)2018-12-07.New sequence ops:cntkx.ops.sequence.padandcntkx.ops.sequence.lengthAdded two new sequence ops.cntkx.ops.sequence.padallows padding on the sequence axis andcntkx.ops.sequence.lengthcalculates the length of the sequence.2018-12-05.Mixture Density NetworkMixture density networks are neural networks that can in principle represent arbitrary conditional
probability distributions in the same way that a conventional neural network can represent arbitrary functions.
MDN are very useful when you need to map an input to several correct targets (aka. one-to-many problem).Updated with Gaussian Mixture Density Network ops and loss function. Ops will allow you to extract mdn coefficients and sample from the network.More details on mdn can be found in thispaperby Christopher Bishop.import cntk as C
import cntkx as Cx

input_tensor = C.input_variable(1, name="input_tensor")
target_tensor = C.input_variable(1, name="target_tensor")

# model
inner = Dense(50, activation=C.relu)(input_tensor)
inner = Dense(50, activation=C.relu)(inner)
prediction_tensor = Dense((ndim + 2) * nmix, activation=None)(inner)

sampled = Cx.sample_gaussian_mdn(prediction_tensor, nmix, ndim)  # sampling node
loss = Cx.gaussian_mdn_loss(prediction_tensor, target_tensor, nmix=nmix, ndim=ndim)  # loss function