Reviewerly/exHarmony huggingface dataset:
Enhancing Efficiency and Quality of Image Caption Generation with CARU

Image caption is textual explanation automatically generated by a computer according to the content in an image. It involves both image and natural language processing, and thus becomes an important research topic in pattern recognition. Deep learning has been successful in accomplishing this task, and the quality of captions generated by existing methods is already high. However, due to the broadness and variety of image caption applications, the current generated captions are still not sufficiently detailed, and the training efficiency can also be improved. Therefore, under the encoder-decoder framework of deep learning, how to use fewer parameters to improve the training efficiency and retain the quality of the generated image descriptions is a huge challenge. In this work, we introduce an improved method based on the encoder-decoder structure, adding an attention mechanism, and applying the content adaptive recurrent unit (CARU), as the decoder, to generate image captions. Inspired by GRU, CARU is designed to have comparable performance with fewer parameters, and is sensitive to the features in hidden layers. The experimental results show, based on MsCOCO dataset, the proposed method achieved better performance than that using GRU as the decoder, and took less training time, effectively improves the training efficiency.
Joint Attention Mechanism Feature Selection for Single Image Reflection Separation

Separating the reflective component from a single reflected image has long been an essential but challenging task. To solve the single image reflection separation problem, we combine the reflection model with deep learning, and design a single image reflection separation network based on a nonlinear reflection image mixing model. The network proposed in this paper consists of a joint attention mechanism for learning multidimensional features, and an encoder and a corresponding three-branch decoder. The three feature decoder modules have the same structure but different weight parameters to perform feature selection and decoding on each component in the reflection image, and step through the task of reflection image separation. The separation network structure further considers the performance of different reflection types. Experimental results on synthetic public datasets of three different reflection types and a real public dataset show that the method proposed in this paper can effectively separate the reflection components in multiple types of reflection images.
Learning Omnidirectional Flow in 360$$^\circ $$ Video via Siamese Representation

Optical flow estimation in omnidirectional videos faces two significant issues: the lack of benchmark datasets and the challenge of adapting perspective video-based methods to accommodate the omnidirectional nature. This paper proposes the first perceptually natural-synthetic omnidirectional benchmark dataset with a 360 $$^\circ $$ field of view, FLOW360, with 40 different videos and 4,000 video frames. We conduct comprehensive characteristic analysis and comparisons between our dataset and existing optical flow datasets, which manifest perceptual realism, uniqueness, and diversity. To accommodate the omnidirectional nature, we present a novel Siamese representation Learning framework for Omnidirectional Flow (SLOF). We train our network in a contrastive manner with a hybrid loss function that combines contrastive loss and optical flow loss. Extensive experiments verify the proposed framework’s effectiveness and show up to 40% performance improvement over the state-of-the-art approaches. Our FLOW360 dataset and code are available at https://siamlof.github.io/ .
Self-supervised Wide Baseline Visual Servoing via 3D Equivariance

One of the challenging input settings for visual servoing is when the initial and goal camera views are far apart. Such settings are difficult because the wide baseline can cause drastic changes in object appearance and cause occlusions. This paper presents a novel self-supervised visual servoing method for wide baseline images which does not require 3D ground truth supervision. Existing approaches that regress absolute camera pose with respect to an object require 3D ground truth data of the object in the forms of 3D bounding boxes or meshes. We learn a coherent visual representation by leveraging a geometric property called 3D equivariance—the representation is transformed in a predictable way as a function of 3D transformation. To ensure that the feature-space is faithful to the underlying geodesic space, a geodesic preserving constraint is applied in conjunction with the equivariance. We design a Siamese network that can effectively enforce these two geometric properties without requiring 3D supervision. With the learned model, the relative transformation can be inferred simply by following the gradient in the learned space and used as feedback for closed-loop visual servoing. Our method is evaluated on objects from the YCB dataset, showing meaningful outperformance on a visual servoing task, or object alignment task with respect to state-of-the-art approaches that use 3D supervision. Ours yields more than 35% average distance error reduction and more than 90% success rate with 3cm error tolerance.
BRENDA Digital Tours: Designing a Gamified Augmented Reality Application to Encourage Gastronomy Tourism and Local Food Exploration

Local food is an essential component of the appeal of a destination for those who wish to experience culinary traditions and gain deep and authentic sense of a place. Within this context, the “BRENDA: Digital Tours” research project features the development of an urban game that focuses on the gastronomic and historical exploration of the Prefecture of Kilkis in Greece. It consists of a gaming environment that supports transmedia storytelling implemented via a fully-featured scenario that enables both virtual and actual exploration, augmented reality gaming, learning of local culture, recipes, and actual food tasting in connection with real-life businesses. This serious game is designed in an adaptive and expandable manner enabling scalable deployment while the end-user is presented with both a desktop-based and portable gaming experience. Each point of interest featured consists of multimedia content and linked gaming components including quizzes and augmented-reality missions. Their completion earns players virtual coins that can be used to purchase cooking ingredients which can be used to unlock and reveal local recipes, while they advance in the classes of the game. Completion of a recipe reveals the full recipe while it offers players the choice to visit a partnered local business and taste the recipe in situ, obtaining extra gaming points.
From Eye to Brain: A Proactive and Distributed Crowdsensing Framework for Federated Learning

Massive amounts of high-quality data are the prerequisite and support for AI technologies. Due to the nature of privacy-preserving and low communication overheads, federated learning (FL) has garnered considerable attention in comparison with traditional data collection methods. However, the performance of FL is hampered by the lack of interested clients and limited local data due to selfishness and individual behavioral preferences. To this end, we propose PractFL, a proactive and distributed framework that incorporates the concept of mobile crowdsensing into the FL paradigm. Specifically, we design an incentive mechanism in the form of virtual red packets, which are a widely used way of monetary reward and gift-giving in social lives. In this article we extend this further by giving meaning to the locations, i.e., the red packets are only accessible at specific places. The virtual red packets’ locations and monetary amounts can be dynamically updated by the cloud center to encourage clients to collect additional data that may benefit the FL process. Further, we propose a distributed behavioral decision engine based on multiarmed bandits (i.e., choose which red packet to go for) in response to the incentive mechanism enforced by the cloud. Considering the movement cost and conflicts with other clients, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula> -anonymity and probabilistic selection are introduced in the distributed behavioral decision to recommend the optimal red packet choice for clients without revealing their privacy. The experimental results demonstrate that PractFL outperforms the baselines in terms of classification accuracy. We also find that PractFL can effectively alleviate the overfitting problem caused by class imbalance during the training.
Application of Participatory Design Methodology in AR: Developing Prototypes for Two Context Scenarios

Recent technological developments and projects are now becoming more multidisciplinary. However, in designing augmented reality training and support systems (ARTSS), researchers in this field may already hold biases that lean towards the de facto standard of what constitutes good ARTSS. These biases are not inherently harmful, but there exists the risk that these may not lead to answering the demands and goals of the intended end-user that are of completely different specializations. To address this risk, we suggest using Par-ticipatory Design (PD) as a methodology for creating and designing ARTSS. By putting the collaborators as co-designers in the authoring of augmented reality prototypes, we can verify that the final ARTSS meets the end-user's requirements.
A Highly Secure Authentication Module for Smart Door Lock with Temporary Key Function

A significant feature of smart door lock systems is the Temporary Key Function (TKF) for visitors. However, existing TKFs in smart door lock systems in use are either vulnerable to attacks or inconvenient for use. This paper thereby proposes elliptic curve cryptography (ECC)-based innovative authentication module for smart door lock systems which enables highly secure and convenient TKF. Based on the authentication module, two protocols are designed for the host and visitor respectively to unlock the door lock system. Security features for the two protocols are verified via Gong Needham Yahalom (GNY) logic. Besides, the proposed prototypes are realized, and the simulation experiments are carried out to study the performance of the protocols. The results show that our scheme is secure and efficient.
AnimeCeleb: Large-Scale Animation CelebHeads Dataset for Head Reenactment

We present a novel Animation CelebHeads dataset (AnimeCeleb) to address an animation head reenactment. Different from previous animation head datasets, we utilize a 3D animation models as the controllable image samplers, which can provide a large amount of head images with their corresponding detailed pose annotations. To facilitate a data creation process, we build a semi-automatic pipeline leveraging an open 3D computer graphics software with a developed annotation system. After training with the AnimeCeleb, recent head reenactment models produce high-quality animation head reenactment results, which are not achievable with existing datasets. Furthermore, motivated by metaverse application, we propose a novel pose mapping method and architecture to tackle a cross-domain head reenactment task. During inference, a user can easily transfer one's motion to an arbitrary animation head. Experiments demonstrate an usefulness of the AnimeCeleb to train animation head reenactment models, and the superiority of our cross-domain head reenactment model compared to state-of-the-art methods. Our dataset and code are available at .
A Design and Development of Internet of Things (IoT) System and Learning Activity to Promote Computational Thinking

Computational Thinking (CT), problem-solving and programming skills are important for the students with major of computer education of higher levels education which related to the computing science contents. The instructions must provide the basic contents and learning activities, including Internet of Things (IoT), big-data and programming, as well as Artificial Intelligence (AI) and machine learning. However, these courses present learning difficulties based upon the students' individual skills. Development and integration of the IoT system would offer supporting tools for learning activities and improve the students' computational thinking, problem-solving, and programming skills. This paper presents the design and development of IoT system to support students' computational thinking, problem-solving, and programming skills training through block-based programming and integrated platforms. The IoT system comprises three layers (hardware, middleware, and applications), using existing platforms and devices as well as newly developed. To validate the viability of the proposed pipelined realization, the developed systems were implemented for higher education level students of computer education program. The students' computational-thinking, problem-solving, and programming skills were significantly improved through the proposed learning system throughout the course.
Applying Software Development Risk Taxonomy in Use Case Points Complexity Factor

The studies of software development risk continues to grow. One is the software development risk taxonomy released by Software Engineering Institute (SEI). Unfortunately, no risk management research has been explicitly combined with software size estimation. Therefore, this study aims to apply risk management over 13 technical and eight environmental factors in the Use Case Points (UCP) method. The dataset consists of 345 risk factors by any sources, then mapped and justified by three experts. This mapping generates the risk frequency, which is finally modified by adding it to the origin weight by UCP. The results of the application of risk to environmental factors dominate as much as 76.81 percent compared to technical factors. There are 2 objects that the most influencing risks, such skill and motivation for both the developer team and end-user. For further research, this study still challenges how risk management can be integrated to obtain better accuracy toward software effort estimation.
MRA of Fault-generated High-frequency Transient Signals using Mathematical Morphology. Comparison with Wavelet Transform

Multiresolution analysis (MRA) is a sequence of decompositions of a signal to observe it at multiple resolutions. Therefore, it allows extracting signal features over a wide range of frequency spectrum. In this sense, high-frequency signals generated by electrical faults (short-circuits) in transmission lines can be characterized. The wavelet transform has become the most developed method by researchers for MRA, but this technique suffers from drawbacks that still makes it difficult to implement in real applications. However, a new method based on mathematical morphology has been proving useful in image processing, so this work proposes to develop the MRA of high-frequency signals by kilometer resolution based on fault location. In this regard, several simulations were performed on a power electrical system modeled in software ATP-Draw with a frequency-dependent transmission line model. These events were developed with high sampling frequencies and also consider the scenarios of different fault resistances and inception angles. Finally, the comparison with the wavelet transform is described.
Semantic Driven Energy based Out-of-Distribution Detection

Detecting Out-of-Distribution (OOD) samples in real world visual applications like classification or object detection has become a necessary precondition in today's deployment of Deep Learning systems. Many techniques have been proposed, of which Energy based OOD methods have proved to be promising and achieved impressive performance. We propose semantic driven energy based method, which is an end-to-end trainable system and easy to optimize. We distinguish in-distribution samples from out-distribution samples with an energy score coupled with a representation score. We achieve it by minimizing the energy for in-distribution samples and simultaneously learn respective class representations that are closer and maximizing energy for out-distribution samples and pushing their representation further out from known class representation. Moreover, we propose a novel loss function which we call Cluster Focal Loss(CFL) that proved to be simple yet very effective in learning better class wise cluster center representations. We find that, our novel approach enhances outlier detection and achieve state-of-the-art as an energy-based model on common benchmarks. On CIFAR-10 and CIFAR-100 trained WideResNet, our model significantly reduces the relative average False Positive Rate(at True Positive Rate of 95%) by 67.2% and 57.4% respectively, compared to the existing energy based approaches. Further, we extend our framework for object detection and achieve improved performance.
Arduino-based Dual Axis Solar Tracking System Prototype

Since solar energy is an infinite source of energy, it can be used as a suitable alternative energy source. One of the technological attempts to utilize solar energy is the use of solar panels. A solar panel is a device consisting of a series of solar cells that can convert sunlight energy into electrical energy. The generated electrical energy adjusts the intensity of the received sunlight. However, in its application, most solar cells are statically placed (not following the movement of the sun), where the position of the solar panels only leads to one specific preventing optimal absorption of the intensity of sunlight and resulting electrical energy generation. Therefore, we need a solar tracking system that can automatically control the solar panels to track the movement of the sunlight so that they can absorb the sunlight optimally. The solar tracker created in this research consists of monocrystalline solar panels, LDRs, INA219 sensor, Arduino board, and servo motors. Data collection was carried out for six hours on three consecutive days. According to this study, the greatest difference in power generated by solar panels occurs between 12:00 and 13:00 WIB, with an average value of active solar tracker power of 0.5 W and static solar tracker value of 0.34 W.
Aspect-Based Sentiment Analysis via Virtual Node Augmented Graph Convolutional Networks

Aspect-based sentiment analysis (ABSA) refers to a fine-grained sentiment analysis task aimed at detecting sentiment polarity towards a given aspect. Recently, graph convolutional networks (GCN) integrated with dependency trees have achieved related appealing results in ABSA. Nevertheless, most existing models fail to preserve the information of the whole graph although global information can often significantly improve their performance. To address this problem, a novel virtual node augmented graph convolutional network (ViGCN) is proposed to further enhance the performance of GCNs in the ABSA task by adding a virtual node to the graph. The virtual node can connect to all the nodes in the graph to aggregate global information from the entire graph and then propagate it to each node. In particular, we construct edges between the virtual node and other nodes based on affective commonsense knowledge from SenticNet and the semantic-relative distances between contextual words and the aspect, effectively enhancing the collected global information towards the given aspect. Extensive experiments on three benchmark datasets illustrate that the ViGCN model can beat state-of-the-art models, proving its effectiveness.
Adversarial training with informed data selection

With the increasing amount of available data and advances in computing capabilities, deep neural networks (DNNs) have been successfully employed to solve challenging tasks in various areas, including healthcare, climete, and finance. Nevertheless, state-of-the-art DNNs are susceptible to quasi-imperceptible perturbed versions of the original images - adversarial examples. These perturbations of the network input can lead to disastrous implications in critical areas where wrong decisions can directly affect human lives. Adversarial training is the most efficient solution to defend the network against these malicious attacks. However, adversarial trained networks generally come with lower clean accuracy and higher computational complexity. This work proposes a data selection (DS) strategy to be applied in the mini - batch training. Based on the cross-entropy loss, the most relevant samples in the batch are selected to update the model parameters in the backpropagation. The simulation results show that a good compromise can be obtained regarding robustness and standard accuracy, whereas the computational complexity of the backpropagation pass is reduced.
A Data Mining Approach to Finding Face Mask from Bangladeshi News Channel

Face Mask Detection is currently a hot topic that has piqued the interest of researchers all over the world. Today, the entire world is dealing with the COVID-19 pandemic. To control the spread of the Coronavirus the most important task people need to do is use a mask. There is still a lot of research and study being done on COVID-19. Several studies have also shown that wearing a face mask significantly reduces the problem of viral transmission. In addition, a person wearing a face mask perceives a sense of protection. When we are at home, we take care of everything, but when we are in public places such as offices, malls, and colleges, it becomes more difficult to keep people safe. Machine Learning and Data Mining are a collection of technologies that provide effective solutions to complex problems in a variety of fields. We attempted to develop a face mask recognition system using machine learning in order to prevent the spread of the Coronavirus. This is a good system for detecting a face mask in news channel images and videos. It can recognize both Mask and No Mask faces. With the advancement of this system, it will be possible to detect whether or not a person is wearing a face mask. If the person is not wearing a face mask, it will display a message such as "No Mask," otherwise it will display "Mask Detected."
Automatic Pruning Rate Derivation for Structured Pruning of Deep Neural Networks

To compress the neural network model, structured pruning has been proposed. However, finding a proper pruning rate to suppress the accuracy degradation of pruned models is difficult because existing structured pruning methods assign the pruning rate manually. As described herein, we propose an automatic pruning rate derivation method for structured pruning to reduce the workload of inefficient manual pruning rate assignment. The value of the pruning error (L1-norm of the pruned weight) depends on the pruning rate. Therefore, to derive the pruning rate, our method compares the pruning error and the threshold. When the pruning error is less than the threshold, the degradation of the pruned model accuracy is suppressed. We demonstrate the superiority of our proposed method over state-of-the-art methods on CIFAR-10 and ImageNet using various ResNets. For example, the proposed method reduces 56.2% parameters of ResNet-50 with similar accuracy of 75.32% to earlier works on ImageNet.