Reviewerly/exHarmony huggingface dataset:
Automatic Grayscale Image Colorization using a Deep Hybrid Model

Image colorization is an interesting yet challenging task due to the descriptive nature of getting a natural-looking color image from any grayscale image. To tackle this challenge and also have a fully automatic procedure, we propose a Convolutional Neural Network (CNN)-based model to benefit from the impressive ability of CNN in the image processing tasks. To this end, we propose a deep-based model for automatic grayscale image colorization. Harnessing from convolutional-based pre-trained models, we fuse three pre-trained models, VGG16, ResNet50, and Inception-v2, to improve the model performance. The average of three model outputs is used to obtain more rich features in the model. The fused features are fed to an encoder-decoder network to obtain a color image from a grayscale input image. We perform a step-by-step analysis of different pre-trained models and fusion methodologies to include a more accurate combination of these models in the proposed model. Results on LFW and ImageNet datasets confirm the effectiveness of our model compared to state-of-the-art alternatives in the field.
A brief survey on Capsule Network

Capsule networks(CapsNets) are new kinds of network representation in deep learning. They are proposed to overcome the shortcomings of convolutional neural networks(CNNs). CNNs do not consider the important spatial level correlation between simple and complex objects. Besides, the pooling operations lose too much spatial information. In contrast, CapsNets use a new architecture that mimics the human visual system to obtain equivariance, instead of the original translational invariance, so that they can use less data to get more extensive generalization in different perspectives. However, as a new research field, the lack of knowledge and the working principle of capsules are still obstacles for researchers to make big progress. In this paper, we review the related works of CapsNets including the structure design, routing mechanisms and applications. Finally, we discuss and summarize the advantages and disadvantages of CapsNets at present, and look forward to the future research directions.
Automation on Washing Machine Using Fuzzy Logic Controller Provided with Three Input and Two Output for Setting the Temperature of Water

Washing machines are household devices that are used to wash clothes and at present it is preferred by all people. Traditional controllers have been proved to be less capable of controlling these devices and therefore, a need has arisen to apply the Fuzzy logic for such controller applications. Fuzzy logic is computer-based approach on “degrees of truthiness” rather than the usual Boolean logic on which computing is based on. It enables system designers to approach the imprecise and unambiguous problems. In this project we have proposed a washing machine controller taking in three inputs to give exact wash time and water temperature for the corresponding inputs. The objective is to minimize the time taken, the electricity consumption and water usage for washing clothes. The principle of this process is to subject input to Fuzzy arithmetic which in turn returns the value of the temperature of water and washing time.
Assessment of traffic congestion with high-resolution remote sensing data and deep convolution neural network

In this study, high resolution (HR) images, GIS and deep convolution neural network (DCNN) are used for assessment of traffic congestion. A DCNN architecture comprises one convolution layer, two pooling layers and a five-layer fully connected neural network (FCNN) evaluated for identifying vehicles in a movable window in HR images. A simple mathematical method is followed for changing the scale and orientation of the movable window to optimally mask and measure the area of vehicles. A formula is appraised to compute the traffic density using the estimated vehicles and road areas. A threshold is used to estimate traffic congestion from the measured traffic density. The method is validated by applying on World View-2 pan-sharpened multispectral images having spatial resolution 0.46 m. In comparison to CNN and ResNet-18, the proposed approach achieves a quite promising accuracy (99%) and needs less training and processing time for measuring traffic congestion in HR images.
Sign Language Recognition by Means of Common Spatial Patterns

Currently, and despite the efforts that have been made, people with hearing impairments often have difficulties to use applications that have been designed for people who can hear, or simply to communicate with their environment. In this work, we present an Argentinian Sign Language (LSA) recognition system which distinguishes between different signs using hand landmarks extracted from the videos of the dataset. The Common Spatial Patterns (CSP) algorithm is used to extract features, and the classification is performed with multiple classifiers. Different experiments have been made from which promising results have been obtained.
Dynamically Mixed Group Convolution to Lighten Convolution Operation

Convolution operation is the most significant component in convolution neural networks (CNNs). However, the high computation cost limits its application on mobiles and embedded devices. To address this problem, in this paper, we develop a dynamically mixed group convolution operation (DMGConv) to lighten convolution operation. It consists of three mixed primary groups, and each primary group includes G dynamical tiny groups. The mixed primary groups not only compress the computation but also can lead to better efficiency. In addition, instead of manual setting G, we dynamically calculate the greatest common divisor of the input channels and output channels of every primary group as the number of G, which improves the self-adaptability of tiny group convolution. To demonstrate our proposed method's efficiency, we compare its parameters and computation with the popular convolutions. Moreover, we utilize our proposed approach to replace the convolution operation in the state-of-the-art CNNs. We perform them on the benchmark classification dataset CIFAR10. Experiment shows that our proposed approach is more lightweight for large input channel size and output channel size and our method can effectively reduce convolution computation and CNN's model size.
Multi-scale selective image texture smoothing via intuitive single clicks

Existing texture filtering techniques acquiesce in the removal of all undesired background texture patterns simultaneously, and lead to the disappearance of small-scale foreground structures. Differently, this paper considers both multi-scale structures and specified texture patterns within an image as significant image contents, and presents a content-aware texture filtering approach via an easy-to-use interactive interface, named intuitive single click. The user’s effort on content-aware texture filtering is reduced to only locating a single click, where it is intuitively considered as one type of texture patterns. This simple interface is made possible by first generating three-scale intermediate filtered images (i.e., high, medium and low) and then performing breadth-first search (BFS) to identify one or more texture regions. Specifically, we utilize BFS to purify image edges in the medium-scale filtered image, on top of which a mask representing texture and non-texture regions is abstracted through the intuitive single-click texture locating and BFS. The mask is applied to effectively maintain multi-scale structures in the low-scale filtered image, and guides to fully smooth textures in the high-scale filtered image. Finally, a clean image is yielded by simply synthesizing the results of two mask operations in the low- and high-scale filtered images. We illustrate by extensive experiments that our approach advances the state-of-the-arts in terms of both the image cleanness and content-preserving ability. Our code is publicly available.
Deep Learning Framework for Face Mask Detection

COVID-19, a pandemic disease spread caused by the corona virus, has taken a toll on our lives. Wearing face masks, sanitizing, and maintaining social distance has been the new normal since the outbreak. As the Corona virus enters our bodies through the mouth and nose, covering those areas can help us to protect ourselves from the virus to some extent. Although vaccination and medications are issued for this disease, WHO states that wearing masks and following Covid protocols should be continued. After the first wave has reached to a control and vaccination started, public has started neglecting face mask and social distancing. This resulted in a massive spread of the second wave, over which some populated countries are losing control. Henceforth, wearing a mask will highly help in preventing the transmission of the disease. This project helps the detection of faces with and without mask from live video streams. By making use of packages like TensorFlow, SciPy, OpenCV, Keras the implementation is done. The model used for face detection is MTCNN. The overall idea of developing this project is to implement an embedded model, which can detect faces from video streams and generate a valid classification of faces with/without mask. This model will be useful for the public surveillances and insist people to put on their masks and prevent the spread of dangerous corona virus. The proposedmodel has obtained a classification accuracy of ~97.2%.
Identification of the nonlinear systems based on the kernel functions

Abstract Constructing an appropriate membership function is significant in fuzzy logic control. Based on the multi‐model control theory, this article constructs a novel kernel function which can implement the fuzzification and defuzzification processes and reflect the dynamic quality of the nonlinear systems accurately. Then we focus on the identification problems of the nonlinear systems based on the kernel functions. Applying the hierarchical identification principle, we present the hierarchical stochastic gradient algorithm for the nonlinear systems. Meanwhile, the one‐dimensional search methods are proposed to solve the problem of determining the optimal step sizes. In order to improve the parameter estimation accuracy, we propose the hierarchical multi‐innovation forgetting factor stochastic gradient algorithm by introducing the forgetting factor and using the multi‐innovation identification theory. The simulation example is provided to test the proposed algorithms from the aspects of parameter estimation accuracy and prediction performance.
Handling Non-Local Executions to Improve MapReduce Performance Using Ant Colony Optimization

Improving the performance of MapReduce scheduler is a primary objective, especially in a heterogeneous virtualized cloud environment. A map task is typically assigned with an input split, which consists of one or more data blocks. When a map task is assigned to more than one data block, non-local execution is performed. In classical MapReduce scheduling schemes, data blocks are copied over the network to a node where the map task is running. This increases job latency and consumes considerable network bandwidth within and between racks in the cloud data centre. Considering this situation, we propose a methodology, “improving data locality using ant colony optimization (IDLACO),” to minimize the number of non-local executions and virtual network bandwidth consumption when input split is assigned to more than one data block. First, IDLACO determines a set of data blocks for each map task of a MapReduce job to perform non-local executions to minimize the job latency and virtual network consumption. Then, the target virtual machine to execute map task is determined based on its heterogeneous performance. Finally, if a set of data blocks is transferred to the same node for repeated job execution, it is decided to temporarily cache them in the target virtual machine. The performance of IDLACO is analysed and compared with fair scheduler and Holistic scheduler based on the parameters, such as the number of non-local executions, average map task latency, job latency, and amount of bandwidth consumed for a MapReduce job. Results show that IDLACO significantly outperformed the classical fair scheduler and Holistic scheduler.
Pase: Parallelization Strategies for Efficient DNN Training

Training a deep neural network (DNN) requires substantial computational and memory requirements. It is common to use multiple devices to train a DNN to reduce the overall training time. There are several choices to parallelize each layer in a DNN. Exhaustively searching this list to find an optimal parallelization strategy is prohibitively time consuming and impractical. The standard practice is to use data parallelism because of its simplicity. However, data parallelism is often suboptimal, and suffers from poor performance and high memory requirement. Expert-designed strategies have been proposed on a case-by-case basis using domain specific knowledge. These expert-designed strategies do not generalize well to DNNs other than the ones for which they were designed, and are not always necessarily the best choice. In this paper, we propose an approach to automatically find efficient parallelization strategies for DNNs from their computation graphs. We present an efficient algorithm to compute these strategies within a reasonable time in practice. We evaluate the effectiveness of our approach on various DNNs. We also compare the performance of the strategies identified by our approach against data parallelism, expert-designed strategies, and the state-of-the-art approaches. Our results show that the strategies found using our approach outperform the baseline data parallelism strategy in all the cases. In addition, our strategies achieve better performance than the expert-designed strategies and the state-of-the-art approaches.
An improved Reversible Data Hiding for Encrypted Images over Cloud

Recently, Reversible Data Hiding produced notable contributions to the data security and privacy protection of encrypted images in cloud environment. This inspired the research topic namely Reversible Data Hiding in Encrypted Images (RDHEI) which allows embedding of auxiliary data/meta data into an already encrypted image so that the original content of the image is privileged. Earlier works on RDHEI show less efficiency in embedding capacity because encrypted images show inadequacy in possessing redundant space. The motivation of the proposed work is to fully exploit the available space in encrypted images for maximal data embedding. In this paper, an improved fully separable RDHEI is proposed using Effective Block Scrambling (EBS) and Overlapping Pixel Value Ordering (OPVO) to improve embedding capacity of encrypted images. The proposed scrambling technique is based on chaotic map and block wise XOR encryption. After the encryption, the data hider performs embedding by dividing the encrypted image into nonuniform blocks followed by three-mode application of Overlapping Pixel Value Ordering. At receiver end, original image can be extracted with three possible cases upon the availability of secret keys. Experimental results emphasize the efficacy of the proposed method which prevails over the state-of-the-art methods.
Depth-Aware Object Tracking With a Conditional Variational Autoencoder

Object tracking is a fundamental task in computer vision and artificial intelligence. However, state-of-the-art object tracking approaches are still prone to failures and are imprecise when applied to challenging scenarios, and their results are generally confidence agnostic. An imprecise deterministic output with low confidence may lead to disastrous consequences and a lack of proof for subsequent operations and human interventions. Deep network training with ambiguous data or the noise inherent in observations (i.e., data uncertainty or aleatoric uncertainty) will result in inherent uncertainties in predictions. In this paper, we exploit probabilistic depth-aware object tracking with a conditional variational autoencoder (CVAE). First, we build a bridge between the Siamese network and the variational autoencoder conditioned with depth images and propose a novel multimodal Bayesian object tracking method. Second, our proposed method yields a complete probability distribution that enables the production of multiple plausible features. Third, the variational autoencoder conditioned by depth images encodes a low-dimensional latent space that conducts depth-aware tracking, which has obvious advantages for challenging tracking scenarios. Our proposed tracking method outperformed the state-of-the-art trackers on the VOT 2016, VOT 2018, and VOT 2019 datasets.
Dense stereo matching in depth discontinuity areas based on edge detection and nearest valid point search

Dense stereo matching based on two viewpoint images is an important step in 3D reconstruction. In this study, a new disparity estimation algorithm for depth discontinuity areas is proposed. Dense stereo matching is implemented on the basis of the semi-global matching algorithm. If the ground truth is known, then mismatched pixels are selected as those with a difference of more than one pixel from the ground truth. Otherwise, the left-right consistency checking process is used. Then, the Canny operator is used to detect image edges. For each mismatched pixel, a maximum total of eight nearest edge points exists in four pairs of symmetric directions. Basically, the process of searching for the nearest valid point proceeds along directions in the following sequence: horizontal, vertical, 45° oblique and 135° oblique. Specifically, for each pair of symmetric directions, the nearest valid pixel is searched along the direction of the distant edge point. If the disparity is successfully reassigned by the nearest valid pixel found, the searching process is terminated. Comparative results obtained using well-known benchmark datasets and unmanned aerial vehicle stereo images demonstrate the excellent performance of the proposed method.
Query Games in Databases

Database tuples can be seen as players in the game of jointly realizing the answer to a query. Some tuples may contribute more than others to the outcome, which can be a binary value in the case of a Boolean query, a number for a numerical aggregate query, and so on. To quantify the contributions of tuples, we use the Shapley value that was introduced in cooperative game theory and has found applications in a plethora of domains. Specifically, the Shapley value of an individual tuple quantifies its contribution to the query. We investigate the applicability of the Shapley value in this setting, as well as the computational aspects of its calculation in terms of complexity, algorithms, and approximation.
DeepObjStyle: Deep Object-based Photo Style Transfer

One of the major challenges of style transfer is the appropriate image features supervision between the output image and the input images (style and content). An efficient strategy would be to define an object map between the objects of the style and the content images. However, such a mapping is not well established when there are semantic objects of different types and numbers in the style and the content images. It also leads to content mismatch in the style transfer output, which could reduce the visual quality of the results. We propose an object-based style transfer approach, called DeepObjStyle, for the style supervision in the training data-independent framework. DeepObjStyle preserves the semantics of the objects and achieves better style transfer in the challenging scenario when the style and the content images have a mismatch of image features. We also perform style transfer of images containing a word cloud to demonstrate that DeepObjStyle enables an appropriate image features supervision. We validate the results using quantitative comparisons and user studies.
Attack Specification Language: Domain Specific Language for Dynamic Training in Cyber Range

Cyber education development is a crucial issue considering the human resource and skill shortage in the current cybersecurity arena. A cyber range is a tried and tested hands-on training in cybersecurity education, providing threat simulation of various scenarios. However, the threat scenario development poses crucial challenges that hurt the learning process and trainee's engagement in training. Firstly, the threat scenarios are static and have limited applicability. Secondly, due to the lack of proper representation of procedures and training scenarios used in attacks, it is hard to recognize redundant procedures. We propose an Attack Specific Language (ASL) based on the Mitre ATT&CK framework. It provides one representation for all threat scenarios. This language will give information about attack techniques in compact ways, which will streamline and automate the cyber range functions of threat and challenge execution. It will help identify and reduce redundancy. ASL will also provide training customization through dynamic threat execution, which will be trainee-aware and will consider the trainee's performance while executing scenarios. It will provide trainees, better engagement, and training experience.
Relation-Aware Fine-Grained Reasoning Network for Textbook Question Answering

Textbook question answering (TQA) is a task that one should answer non-diagram and diagram questions accurately, given a large context which consists of abundant diagrams and essays. Although lots of studies have made significant progress in the natural image question answering (QA), they are not applicable to comprehending diagrams and reasoning over the long multimodal context. To address the above issues, we propose a relation-aware fine-grained reasoning (RAFR) network that performs fine-grained reasoning over the nodes of relation-based diagram graphs. Our method uses semantic dependencies and relative positions between nodes in the diagram to construct relation graphs and applies graph attention networks to learn diagram representations. To extract and reason over the multimodal knowledge, we first extract the text that is the most relevant to questions, options, and the instructional diagram which is the most relevant to question diagrams at the word-sentence level and the node-diagram level, respectively. Then, we apply instructional-diagram-guided attention and question-guided attention to reason over the node of question diagrams, respectively. The experimental results show that our proposed method achieves the best performance on the TQA dataset compared with baselines. We also conduct extensive ablation studies to comprehensively analyze the proposed method.
Work-in-Progress-Development of Immersive Nursing Skills Learning System and Evaluation of Learning Effectiveness

Most nursing skills training programs involve teacher demonstrations in a professional classroom, and the students follow along step by step. The teachers correct the students' techniques' when they are practicing. However, teachers cannot give feedback to multiple students simultaneously. Thus, students may be restricted by the time and space resulting in a discontinuity in the practice process, and over time, students may forget the steps of the technique. Moreover, students use patient models to practice their skills rather than obtaining real-life experiences. To address these obstacles to effective learning, we developed an “immersive nursing technology learning system” to establish a virtual clinical ward. We trained nursing students to use head-mounted displays and controllers to click, grab, move, and drag items to complete basic actions, construct technical knowledge, and establish familiarity with specific skills. We employed the System Usability Scale to evaluate the usability of the system interface and actual teaching, and the result was an “OK” level. The learning effect of the experimental group's technique was indeed improved. This study can be used as a pilot for immersive nursing technology education. In the future, we hope to expand the nursing technology learning systems to focus on hand movements.