Continuous Kernel Convolution - GeeksforGeeks

14 Aug, 2021
Continuous Kernel convolution was proposed by the researcher of Verije University Amsterdam in collaboration with the University of Amsterdam in a paper titled ‘CKConv: Continuous Kernel Convolution For Sequential Data‘. The motivation behind that is to propose a model that uses the properties of convolution neural networks and Recurrent Neural networks in order to process a long sequence of image data.
Let x ∶R → RNc and ψ ∶ R → RNc be a vector-valued signal and kernel on R, such that x = {xc}Nc and ψ = {ψc} NC c=1. The convolution operation can be defined as:
However, practically the input signal x is gather from sampling. Thus, the input signal and convolution can be defined as  
Input signal:
Convolution:
and the equation that is centered around t is given by: 
Now for continuous kernel convolution, we will use a convolution kernel ψ as continuous function parameterized over a small NN called MLPψ. It takes (t−τ ) as input and outputs the value of the convolution kernel at that position ψ(t−τ ). The continuous kernel can be formulated by:
If the sampling factor is different from training sampling factor, then we can perform convolution operation in following ways: 
 
For the input sequence 
. The recurrent unit is given by:
where U, W, V are the input-to-hidden, hidden-to-hidden and hidden-to-output connections of the unit. h(τ ), y ̃(τ ) depict the hidden representation and the output at time-step τ, and σ represents a point-wise non-linearity.
Now, we unroll the above equation for t steps: 
where h(−1) is the initial state of the hidden representation. h(t) can also be represented in following way: 
The above equation provides us with following conclusion: 
Vanishing gradient and exploding gradient problem in RNN is caused by the term x(t-τ) τ steps back in the past being multiplied with an effective convolution weight ψ(τ )=WτU.
Linear recurrent unit can be defined as the convolution of input and exponential convolution functions.
Let {∆ti=(t − τi)}N i=0 be a sequence of relative positions. The convolution kernel MLPψ is parameterized by a conventional L-layer neural network: 
MLP
where, 
 is used to add non-linearity such as ReLU.
In this implementation, we will be training the CKconv model on the sMNIST dataset, for this implementation, we will be using colaboratory that is provided to us by Google.
Python3
# code credits: https://github.com/dwromero/ckconv# first, we need to clone the ckconv repository from Github! git clone https://github.com/dwromero/ckconv # Now, we need to change the pwd to ckconv directorycd ckconv # Before actually train the model,# we need to first install the required modules and libraries! pip install -r requirements.txt # if the above command fails, please make sure that following modules installed using# command belowpip install ml-collections torchaudio mkl-random sktime wandb # Now to train the model on sMNIst dataset, run the following commands! python run_experiment.py --config.batch_size=64 --config.clip=0 \--config.dataset=MNIST --config.device=cuda --config.dropout=0.1 \--config.dropout_in=0.1 --config.epochs=200 --config.kernelnet_activation_function=Sine \--config.kernelnet_no_hidden=32 --config.kernelnet_norm_type=LayerNorm \--config.kernelnet_omega_0=31.09195739463897 --config.lr=0.001 --config.model=CKCNN \--config.no_blocks=2 --config.no_hidden=30 --config.optimizer=Adam \--config.permuted=False --config.sched_decay_factor=5  --config.sched_patience=20 \--config.scheduler=plateau
Below are the results of the above training of CKCNN on sMNIST data:
CkConv training Result
Ckconv is able to very complex and non-linear function easily.
Contrary to RNNs, CKConvs do not rely on any form of recurrence for considering large memory horizons and have global long-term dependencies.
CKCNNs do not make use of Back-Propagation Through Time(BPTT). Consequently, CKCNNs can be trained in parallel.
CKCNNs can also be deployed at resolutions other than the resolution on which it is trained.
CKConv paper
surindertarika1234
gabaa406
abhishek0719kadiyan
sooda367
gulshankumarar231
Machine Learning
Machine Learning
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
Introduction to Recurrent Neural Network
Support Vector Machine Algorithm
Intuition of Adam Optimizer
CNN | Introduction to Pooling Layer
Convolutional Neural Network (CNN) in Machine Learning
Markov Decision Process
k-nearest neighbor algorithm in Python
Singular Value Decomposition (SVD)
Q-Learning in Python
SARSA Reinforcement Learning