Reviewerly/exHarmony huggingface dataset:
Customized Preview Video Generation Using Visual Saliency: A Case Study with Vision Dominant Videos

Provisioning preview video for long duration videos has always been an expectation. This paper proposes an approach for user-defined customizable preview video generation using visual saliency by extracting some of the fundamental features of the human visual system such as color, intensity and motion. This motivation led to investigating this option for Vision Dominant videos viz., sports videos, wild-life videos, sea world videos etc. Our proposal for visual saliency computation follows the PQFT (Phase Spectrum of Quaternion Fourier transform) model for feature extraction where phase information is used to detect motion(activity)within the frame. The proposed methodology comprises three main stages, with the pre-processing stage that operates on the dataset to reduce the overall computational complexity. The intermediate stage, the saliency detection stage involves feature extraction algorithm to compute saliency values for each frame generating saliency maps and curves that helps in classifying the videos as low, medium and high activity videos. In the final stage, customization is provisioned based on the user's choice of percentage or duration reduction of the original duration of the video to generate preview with keyframe extraction based on maximum saliency values. The PQFT Model deployed is independent of prior knowledge and other parameters making it computationally simple. The experimental results indicate that the proposed method is capable of generating stable and good results. This novel method finds application in different fields towards automatic summarization facilitating preview video on video hosting and streaming websites.
SAMNet: Semantic Aware Multimodal Network for Emoji Drawing Classification

In the current era, the mode of communication through mobile devices is becoming more personalized with the evolution of touch-based input methods. While writing on touch-responsive devices, searching for emojis to capture the true intent is cumbersome. To solve this problem, the existing solutions consider either the text or only stroke-based drawings to predict the appropriate emojis. We do not leverage the full context by considering only a single input. While the user is digitally writing, it is challenging for the model to identify whether the intention is to write text or draw an emoji. Moreover, the model’s memory footprint and latency play an essential role in providing a seamless writing experience to the user. In this paper, we investigate the effectiveness of combining text and drawing as input to the model. We present SAMNet, a multimodal deep neural network that jointly learns the text and image features. Here image features are extracted from the stroke-based drawing and text from the previously written context. We also demonstrate the optimal way to fuse features from both modalities. The paper focuses on improving user experience and providing low latency on edge devices. We trained our model with a carefully crafted dataset of 63 emoji classes and evaluated the performance. We achieve a worst-case On-Device inference time of 60 ms and 76.74% top-3 prediction accuracy with a model size of 3.5 MB. We evaluated the results with the closest matching application-DigitalInk and found that SAMNet provided a 13.95% improvement in the top-3 prediction accuracy.
Class Agnostic, On-Device and Privacy Preserving Repetition Counting of Actions from Videos Using Similarity Bottleneck

We present a practical, privacy-preserving on-device method to get the repetition count of an action in a given video stream. Our approach relies on calculating the pairwise similarity between each sampled frame of the video, using the per frame features extracted by the feature extraction module and a suitable distance metric in the temporal self-similarity(TSM) calculation module. We pass this calculated TSM matrix to the count prediction module to arrive at the repetition count of the action in the given video. The count prediction module is deliberately designed to not pay any attention to the extracted per frame features which are video specific. This self-similarity bottleneck enables the model to be class agnostic and allows generalization to actions not observed during training. We utilize the largest available dataset for repetition counting, Countix, for training and evaluation. We also propose a way for effectively augmenting the training data in Countix. Our experiments show SOTA comparable accuracies with significantly smaller model footprints.
Machine Learning Based Webcasting Analytics for Indian Elections - Reflections on Deployment

India is the largest democracy in the world. Elections play an irreplaceable role in reverberating the voice of its citizens in the governing institutions of this country. To safeguard the power vested in the people, it is essential that the voting process is safe, fair and transparent. This can be very well ensured by effective surveillance of polling activities and analysis of the real-time data that can be gathered from the polling stations. The scattered and widespread locations of the polling stations also requires proper synchronization and organization of the data collection and analysis processes. This paper presents a machine learning-based web analytics system specifically designed to monitor polling stations during the Goa State Assembly elections in 2022. The system accepts as input the CCTV video feeds generated by Internet Protocol (IP) cameras from several polling centres and processes it optimally to produce and transmit analytical information in real-time to the command and control centre. We also highlight the practical challenges faced in deploying this system and their resolution. The successful implementation of this system for the Goa Assembly Elections 2022, first of its kind in India, demonstrates the effectiveness and reliability of this approach.
Novel Image and Its Compressed Image Based on VVC Standard, Pair Data Set for Deep Learning Image and Video Compression Applications

More than 80 percent of online traffic is video and image traffic and this will likely rise in the upcoming years. Images and video have multiple dimensions to grow data rate via increasing frame resolution, frame depth, multi-view representation etc. Thus it is very crucial to compress these images and videos efficiently. Lack of sufficient experimental data is a major setback for the development of image and video compression based on deep learning models. This study presents a new kind of data set for the research community with the goal of advancing the state-of-the-art in image compression using deep learning models. The proposed data set consists of the image and its corresponding VVC (Versatile Video Coding) standard based compressed image as a label of the input image for two quantization parameters. Images from different states of Indian subcontinent area has been captured, containing common objects in their natural context, the beautiful campus of Indian Institute of Technology Madras, which is blessed with rich flora and fauna, and is home to several rare wildlife species, scenes from Himalayas, Clouds in Cherrapunji, Indoor scenes etc. has been captured. The data set will be made publicly to the research community. Statistical analysis of the data set is presented along with VVC compression standard coding analysis.
Low-Textural Image Registration: Comparative Analysis of Feature Descriptors

Industrial machine-vision (MV) applications require high-speed stitching of low-textural images from multiple high-resolution cameras for Field-of-View expansion. The most vital step in the stitching process is the effective and efficient extraction of features, which becomes challenging for low-textural images. This paper presents a comparative study of five popular feature descriptor algorithms for image stitching viz. Scale Invariant Feature Transform (SIFT), Speeded Up Robust Feature (SURF), Oriented Fast and Rotated BRIEF (ORB), Binary Robust invariant scalable keypoints (BRISK), and Accelerated-KAZE (AKAZE). The focus of this paper is to present a study of the performance comparison among these feature extraction methods for low-textural images from real-time steel surface inspection systems. Primarily, synchronized images of steel rolled at room temperatures are obtained from a two-camera network with overlapping regions. Feature descriptor algorithms extract features from two images with an overlapping area and further match the features using K-Nearest Neighbour (KNN) algorithm. The performance of the five feature descriptor algorithms is evaluated using a low-textural dataset that consists of a set of 177 images captured from two cameras placed at a fixed distance from each other. The efficiency of these algorithms is quantitatively and qualitatively evaluated using execution time, sensitivity, and specificity. Finally, this paper provides guidelines for future research on problems with FOV expansion in industrial scenarios.
Multiple Object Tracking Based on Temporal Local Slice Representation of Sub-regions

Multiple object tracking (MOT) involves consistent labeling of objects in a given scene. A scene consists of multiple frames and within each frame rectangular subregions are specified as objects of interest. The task is to label the same object across frames with same identifier. However challenges in this setting involve, change in posture of the object, mild background change in the object region, occlusion, lighting changes, speed of movement and other such critical parameters. MOT is important because of its various applications in mobile robots, autonomous driving, and video surveillance analysis. There a number of neural network based methods which add modules based on property of interest such as a sub-network for velocity, a network for physical motion characteristics and networks based on pixel and edge information characteristics. However they have difficulty dealing with long duration occlusions as well as generalization issues due to millions of parameters and implicit overfitting. We present a new idea called, Temporal Local Slicing (TLS) that obtains local information across frames for a given subregion in the object vectorization step. The vectorization involves histogram of pixel intensities for red, blue and green channels of the sub region. We have performed a total of five experiments and observed the effectiveness of TLS and also a new idea of Gossip vectorization in Multiple object tracking. The object recognition accuracy of TLS vectors is 99.5% and mAP score of 99.1% on train and test partition of a video scene. However the MOT specific scores have been MOTA 56%, IDF1 72%, Recall 56.7%, Precision 98.5% and LOCA 91.9%. These are non-trivial scores indicating potential value in the idea of TLS vectorization.
Fast Detection and Rule Based Classification of Bharatanatyam hasta mudra

Analysis of Bharatanatyam hand gestures extensively requires modern-day computer vision approaches. This paper demonstrates a real-time classification approach to these hand gestures by utilizing the MediaPipe hand model. The process commenced with hand detection and keypoint localization using MediaPipe. The keypoints were utilized to establish a rule for a particular gesture based on joint angle calculation. Real-life data were collected using our model from a well-trained dancer to set different rules. A specific gesture is classified based on the Euclidean distance measure between the joint angles of the gesture and the rule. The gesture exhibiting minimum distance would be the most matched class. The proposed work is also validated by capturing data from three dance learners with different experiences. The captured data achieved an average accuracy of 72.04%. The model took an average of 0.06 s of execution time to classify a gesture in real-time.
Structure-Based Learning for Robust Defense Against Adversarial Attacks in Autonomous Driving Agents

Owing to the recent remarkable advancements in computer vision and artificial intelligence, resilient self-driving cars are now becoming a reality. In this paper, we detail an autonomous car prototype that uses vision-based perception for control and navigation. The car uses a deep neural network to classify traffic symbols based on which motor control decisions are taken. The classification model is tested in various environmental settings and is observed to be robust. In real-world situations, misclassification in autonomous systems can lead to serious accidents. Although the classification model is reliable, there may arise different environmental scenarios where misclassification can occur. In order to simulate such a scenario, we explore the effects of different adversarial attacks on the network. It is observed that the attacks are able to trick the network into misclassifying symbols. Therefore, a network defense mechanism is crucial for handling such scenarios. We present a structure-based learning approach for a robust defense mechanism. The approach is self-supervised and it leverages the use of data augmentations of a sample to have different representations of the same. This ensures that the network is trained to learn the base structure for various environmental settings, thereby nullifying the effect of different adversarial attacks.
Attending Local and Global Features for Image Caption Generation

Attention based deep learning architectures have been used extensively for image captioning. Early methods used attention over global features for generating captions. After the advent of local or object-level features, most researchers started using local features as they provided better performance than global features. Thus the importance of global features in caption generation has been ignored in most current methods. Though the local features provide good performance, they may not adequately capture the underlying relations between different local features. On the contrary, the global features can capture the relationship between different object/local features of the image and may provide essential cues for better caption generation. This work proposes a method that utilizes mean global features to attend to salient local features while generating caption. The global features are extracted using a CNN, and local features are extracted using a Faster RCNN. The proposed decoder utilizes these features to generate the caption. Experiments carried out on the MSCOCO dataset indicate that our approach generates better captions indicating the effectiveness of the proposed method.
A Transformer-Based U-Net Architecture for Fast and Efficient Image Demoireing

Recently, transformer based deep neural networks have been found useful in solving various image restoration tasks like image denoising, deblurring, deraining etc., producing significant improvement in PSNR and SSIM over CNN based techniques on benchmark datasets. These networks have effectively addressed quadratic computational complexity issue with increasing image resolution by making use of novel self attention strategies on local image windows. In this paper, we propose a fast and efficient UNet based architecture using transformer modules for the image demoireing task. The proposed architecture is computationally very efficient as the transformer blocks perform non-overlapping window-based self-attention instead of global self attention. We further improve upon the computational complexity by using decreasing window sizes across scales under the proposed U-Net multi resolution framework. To the best of our knowledge, ours is the first deep network architecture using transformer blocks for the image demoireing problem producing comparable results with state of the art techniques both visually and quantitatively on the CFAMoire challenge dataset [23].
Data-Centric Approach to SAR-Optical Image Translation

Image-to-image translation, especially from SAR to Optical domains, is of great importance in many Earth observation applications. However, the highly complex and unregularised nature of satellite images makes this problem challenging. Existing methods usually work on improving the underlying model to get better results and end up moving towards highly complex and unstable models tested on limited variations of the Earth’s surface. In this paper, we propose a new data-centric approach toward SAR-Optical image translation. The methodology of segregating and categorising the data according different types of land surfaces leads to improvements in image quality and simplifies the task of scaling the model and improving the results. The model is able to effectively capture and translate features unique to different land surfaces and experiments conducted on randomised satellite image inputs demonstrate that our approach is viable in significantly outperforming other baselines.
MTFL: Multi-task Federated Learning for Classification of Healthcare X-Ray Images

Deep learning models have achieved state-of-the-art in many challenging domains, whereas it is a data-hungry method. Collecting sensitive and labelled medical data sets is challenging and costly. Recently, federated learning has been used to train a model without sharing the data at a central place for a single task. We propose a novel Multi-task federated learning (MTFL) approach to utilize the data sets of various similar kinds of tasks. We used two binary class X-ray data sets: Pneumonia disease classification and TB disease classification. We compared MTFL with federated learning for a single task and CNN with data in one place. Results show that MTFL has achieved better specificity and accuracy than other models.
Sequence Recognition in Bharatnatyam Dance

Bharatanatyam is the oldest Indian Classical Dance (ICD) which is learned and practiced across India and the world. Adavu is the core of this dance form. There exist 15 Adavus and 58 variations. Each Adavu variation comprises a well-defined set of motions and postures (called dance steps) that occur in a particular order. So, while learning Adavus, students not only learn the dance steps but also take care of its sequence of occurrences. This paper proposed a method to recognize these sequences. In this work, firstly, we recognize the involved Key Postures (KPs) and motions in the Adavu using Convolutional Neural Network (CNN) and Support Vector Machine (SVM), respectively. In this, CNN achieves 99% and SVM’s recognition accuracy becomes 84%. Next, we compare these KP and motion sequences with the ground truth to find the best match using the Edit Distance algorithm with an accuracy of 98%. The paper contributes hugely to the state-of-the-art in the form of digital heritage, dance tutoring system, and many more. The paper addresses three novelties; (a) Recognizing the sequences based on the KPs and motions rather than only KPs as reported in the earlier works. (b) The performance of the proposed work is measured by analyzing the prediction time per sequence. We also compare our proposed approach with the previous works that deal with the same problem statement. (c) It tests the scalability of the proposed approach by including all the Adavu variations, unlike the earlier literature, which uses only one/two variations.
An End-to-End Fast No-Reference Video Quality Predictor with Spatiotemporal Feature Fusion

This work proposes a reliable and efficient end-to-end No-Reference Video Quality Assessment (NR-VQA) model that fuses deep spatial and temporal features. Since both spatial (semantic) and temporal (motion) features have a significant impact on video quality, we have developed an effective and fast predictor of video quality by combining both. ResNet-50, a well-known pre-trained image classification model, is employed to extract semantic features from video frames, whereas I3D, a well-known pre-trained action recognition model, is used to compute spatiotemporal features from short video clips. Further, extracted features are passed through a regressor head that consists of a Gated Recurrent Unit (GRU) followed by a Fully Connected (FC) layer. Four popular and widely used authentic distortion databases LIVE-VQC, KoNViD-1k, LIVE-Qualcomm, and CVD2014, are utilized for validating the performance. The proposed model demonstrates competitive results with a considerably decreased computation complexity.
DeepTemplates: Object Segmentation Using Shape Templates

Object segmentation is a crucial component for many vision applications like object editing and augmented reality. Traditional pixel-wise segmentation techniques result in irregularities around object boundaries. However, applications like lip makeup require smooth boundaries that resemble a typical lip “template-shape”. We propose an encoder-decoder architecture that is explicitly conditioned to utilize the underlying template-shape of an object for segmentation. The decoder was trained separately to generate a template-shaped segment obtained from landmark points of an object. The encoder is then trained to predict these landmarks using Euclidean loss. Finally, we jointly train the encoder and decoder by incorporating the decoder’s segmentation loss to refine the landmarks, which conditions the network to produce template-shaped object segments. The performance of the proposed method was evaluated with mIOU and f-score measures on the HELEN data set for lip segmentation. We observed perceptually superior segments with smooth object boundaries when compared to state-of-the-art techniques.
Combining Non-local Sparse and Residual Channel Attentions for Single Image Super-resolution Across Modalities

Single image super-resolution (SISR) is an ill-posed problem that aims to generate a high-resolution (HR) image from a single low-resolution (LR) image. The main objective of super-resolution is to add relevant high-frequency detail to complement the available low-frequency information. Classical techniques such as non-local similarity and sparse representations have shown promising results in the SISR task. Nowadays, deep learning techniques such as convolutional neural networks (CNN) can extract deep features to improve the SISR results. However, CNN does not explicitly consider similar information in the image. Hence, we employ the non-local sparse attention (NLSA) module in the CNN framework such that it can explore the non-local similarity within an image. We consider sparsity in the non-local operation by focusing on a particular group named attention bin among many groups of features. NLSA is intended to retain the long-range of non-local operation modeling capacity while benefiting from the efficiency and robustness of sparse representation. However, NLSA focuses on similarity in spatial dimension by neglecting any channel-wise significance. Hence, we try to rescale the channel-specific features adaptively while taking into account channel interdependence by using residual channel attention. We combine the advantages of non-local sparse attention (NLSA) and residual channel attention to produce competitive results in different image modalities such as optical color images, depth maps, and X-Ray without re-training.