Reviewerly/exHarmony huggingface dataset:
MONstEr: A Deep Learning-Based System for the Automatic Generation of Gaming Assets

In recent years, we have witnessed the spread of computer graphics techniques, used as a background map for movies and video games. Nevertheless, when creating 3D models with conventional computer graphics software, it is necessary for the user to manually change the placement and size. This requires expertise of computer graphics architecture and operations, which is time demanding. Applying Artificial Intelligence (AI) to games is currently an established research field. Starting from such premises, in this paper MONstEr (dEEp lEArNiNG GENErAtiON AssEt) a system for the automatic generation of virtual asset for videogames is presented. MONstEr exploits the principle of Deep Learning (DL) and in particular Generative models to automatically design new assets for videogames. The DL pipeline is the core of this system and it is based on a Deep Convolutional Generative Adversarial Network followed by Pixel2Mesh architecture for the 3D models generation. The approach was applied and tested on a newly collected dataset of images, “GameAssetDataset” which comprises characters representation extracted thanks a web crawler algorithm specifically developed for its acquisition. MONstEr expedites the implementation of solutions for new gamining environments, requiring only a small intervention in the 3D construction to insert the object in the game scene.
Enabling Efficient Training of Convolutional Neural Networks for Histopathology Images

Convolutional Neural Networks (CNNs) have gained lots of attention in various digital imaging applications. They have proven to produce incredible results, especially on big data, that require high processing demands. With the increasing size of datasets, especially in computational pathology, CNN processing takes even longer and uses higher computational resources. Considerable research has been conducted to improve the efficiency of CNN, such as quantization. This paper aims to apply efficient training and inference of ResNet using quantization on histopathology images, the Patch Camelyon (PCam) dataset. An analysis for efficient approaches to classify histopathology images is presented. First, the original RGB-colored images are evaluated. Then, compression methods such as channel reduction and sparsity are applied. When comparing sparsity on grayscale with RGB modes, classification accuracy is relatively the same, but the total number of MACs is less in sparsity on grayscale by 77% than RGB. A higher classification result was achieved by grayscale mode, which requires much fewer MACs than the original RGB mode. Our method’s low energy and processing make this project suitable for inference on wearable healthcare low powered devices and mobile hospitals in rural areas or developing countries. This also assists pathologists by presenting a preliminary diagnosis.
On-the-fly syntax highlighting using neural networks

With the presence of online collaborative tools for software developers, source code is shared and consulted frequently, from code viewers to merge requests and code snippets. Typically, code highlighting quality in such scenarios is sacrificed in favor of system responsiveness. In these on-the-fly settings, performing a formal grammatical analysis of the source code is not only expensive, but also intractable for the many times the input is an invalid derivation of the language. Indeed, current popular highlighters heavily rely on a system of regular expressions, typically far from the specification of the language's lexer. Due to their complexity, regular expressions need to be periodically updated as more feedback is collected from the users and their design unwelcome the detection of more complex language formations. This paper delivers a deep learning-based approach suitable for on-the-fly grammatical code highlighting of correct and incorrect language derivations, such as code viewers and snippets. It focuses on alleviating the burden on the developers, who can reuse the language's parsing strategy to produce the desired highlighting specification. Moreover, this approach is compared to nowadays online syntax highlighting tools and formal methods in terms of accuracy and execution time, across different levels of grammatical coverage, for three mainstream programming languages. The results obtained show how the proposed approach can consistently achieve near-perfect accuracy in its predictions, thereby outperforming regular expression-based strategies.
A novel metric learning framework by exploiting global and local information

Distance metric learning plays a significant role in improving the generalization of algorithms related to distance metrics. In this paper, we first propose a generalized Mahalanobis metric learning framework (called GLML) to make use of the prior knowledge from the samples including global and local information, the main goal of which is to enlarge the distance of dissimilar sample pairs and shrink the distance of similar sample pairs simultaneously. Specifically, the metric learning problem is guided by a discriminative regularization through incorporating the pair-wise and class-wise information. Moreover, an effective alternating iterative algorithm is developed to optimize the proposed GLML, and the convergence of the algorithm is demonstrated theoretically. Then a boosting algorithm of GLML (BGLML) is proposed, where the low-rank basis learning is jointly optimized with the metric to better uncover the data structure and mitigate the computational burden. Following that, numerical experiments are carried out on binary classification and multi-classification problems. With different assessment criteria, experiment results from different scale datasets confirm that the proposed methods either improve generalization or have comparable results compared with the state-of-the-art algorithms.
Analysing multi-index overlay and fuzzy logic models for lode-gold prospectivity mapping in the Ahafo gold district – Southwestern Ghana

Mineral prospectivity mapping (MPM) is a multi-criteria decision-making process that identifies and prioritises potential zones for mineral exploration. Multi-index overlay and fuzzy logic models were employed in this paper to conduct GIS-based MPM of the Ahafo gold district, which is located on the northwestern margin of the Sefwi granitegreenstone belt in Ghana. Mineralisation in the area is structurally controlled with a high degree of hydrothermal alteration. Therefore, the mineral systems approach was used to translate the understanding of orogenic gold hydrothermal system into mappable exploration criteria, leading to the generation of nine evidential layers. These layers represented mappable spatial proxies for active fluid pathways (i.e. proximity to NEstriking basement faults: D2 to D3), physical traps (i.e. proximity to N- to NNE-strike sinistral fault: D4; proximity to litho-contacts; proximity to the intersection of N- to NNE- and NE-strike faults; density of N- to NNE- and NE-strike faults; litho-competency contrast), and chemical traps (i.e. proximity to Fe-rich dykes; litho-reactivity contrast; proximity to hydrothermal alteration zone), all of which are critical for hydrothermal fluid migration, focusing, and gold deposition. Weights and confidence factors were assigned to the evidential layers in the fuzzy logic approach based on experts' knowledge of how the layers are relevant to orogenic gold mineralising processes. In contrast, weights were assigned to layers in the multi-index overlay model based on prediction-area (P-A) plots and normalised densities, limiting the subjectivity of assigning weights to the layers. In order to delineate potential zones for lode-gold mineralisation in the study area, the weighted evidential layers were combined to create multi-index overlay and fuzzy prospectivity maps. The continuous prospective scores of the prospectivity maps were discretised and classified using a concentration-area (C-A) fractal model. The resulting discretised prospectivity maps based on the location of 18 known lode-gold occurrences (i.e. deposits and economic prospects) were further evaluated using P-A plots. The discretised multi-index overlay prospectivity map predicted 76% of known lode-gold occurrences, with 24% of the study area as prospective, whereas the discretised fuzzy prospectivity map predicted 74% of known lode-gold occurrences, with 26% of the study area as prospective. The high potential zones identified on the prospectivity maps were spatially located in areas with structural complexity along: (i) NE-SW structural trends (i.e. 40°–50°); and (ii) secondary structural trends in the NS to NNE-SSW direction (i.e. 0°–10° and 15°–25°). These structural trends control lode-gold mineralisation in the Ahafo gold district..
Evaluating the benefits of empowering model‐driven development with a machine learning classifier

Abstract Increasingly, the model driven engineering (MDE) community is paying more attention to the techniques offered by the machine learning (ML) community. This has led to the application of ML techniques to MDE related tasks in hope of increasing the current benefits of MDE. Nevertheless, there is a lack of empirical experiments that evaluate the benefits that ML brings to MDE. In this work, we evaluate the benefits of empowering model engineers of model‐driven development (MDD) with an ML classifier. To do this, we tackled how to embed the ML classifier as part of the MDD. Then, this was evaluated using two different real industrial cases. Our results show that despite the ML part takes an extra effort, the use of the ML classifier pays off in terms of the quality results, the perceived usefulness, and intention to use.
“Do not deceive me anymore!” interpretation through model design and visualization for instagram counterfeit seller account detection

Counterfeit goods sold online have amounted to $330 billion in losses worldwide. Of these, luxury brand losses account for $30.3 billion. The social and economic effects of counterfeit goods have become more severe with the shift of counterfeit sales channels to social networking services (SNSs) since the first decade of the 2000s when these networks became active. In particular, the number of counterfeit sellers on Instagram increased by more than 171% in 2019. This study endeavors to detect counterfeit seller accounts in the SNS environment through a deep learning method that implements an algorithm to differentiate legitimate and counterfeit seller accounts. We designed a model to classify counterfeit seller accounts and general accounts using image and textual data collected from Instagram. As a result, the proposed model obtained a final account detection accuracy of 100% and demonstrated the possibility of use as a counterfeit seller account detector. Moreover, the elements that influenced the results and the parts that the model focused on were identified using a visual analysis method to improve the interpretation capability of the algorithm and explain the results. The difference between human and machine judgments was analyzed based on the visualization.
Chaotic neural networks and farfalle construction based parallel keyed secure hash function

Abstract Parallel computing of hash functions along with the security requirements have great advantage in order to reduce the time consumption and overhead of the CPU. In this article, a keyed hash function based on farfalle construction and chaotic neural networks (CNNs) is proposed, which generates a hash value with arbitrary (defined by user) length (eg, 256 and 512 bits). The proposed hash function has parallelism merit because it is built over farfalle construction which avoids the dependency between the blocks of a given message. Moreover, the proposed hash function is chaos based (ie, it relies on chaotic maps and CNNs which have non‐periodic behavior). The security analysis shows that the proposed hash function is robust and satisfies the properties of hash algorithms, such as random‐like (non‐periodic) behavior, ideal sensitivity to original message and secret key, one‐way property and optimal diffusion effect. The speed performance of the hash function is also analyzed and compared with a hash function which was built based on sponge construction and CNN, and compared with secure hash algorithm (SHA) variants like SHA‐2 and SHA‐3. The results have shown that the proposed hash function has lower time complexity and higher throughput especially with large size messages. Additionally, the proposed hash function has enough resistance to multiple attacks, such as collision attack, birthday attack, exhaustive key search attack, preimage and second preimage attacks, and meet‐in‐the‐middle attack. These advantages make it ideal to be used as a good collision‐resistant hash function.
Deep Learning-Based Construction and Processing of Multimodal Corpus for IoT Devices in Mobile Edge Computing

Dialogue sentiment analysis is a hot topic in the field of artificial intelligence in recent years, in which the construction of multimodal corpus is the key part of dialogue sentiment analysis. With the rapid development of the Internet of Things (IoT), it provides a new means to collect the multiparty dialogues to construct a multimodal corpus. The rapid development of Mobile Edge Computing (MEC) provides a new platform for the construction of multimodal corpus. In this paper, we construct a multimodal corpus on MEC servers to make full use of the storage space distributed at the edge of the network according to the procedure of constructing a multimodal corpus that we propose. At the same time, we build a deep learning model (sentiment analysis model) and use the constructed corpus to train the deep learning model for sentiment on MEC servers to make full use of the computing power distributed at the edge of the network. We carry out experiments based on real-world dataset collected by IoT devices, and the results validate the effectiveness of our sentiment analysis model.
3DeformR

3D objects are the primary media in virtual reality environments in immersive cyberspace, also known as the Metaverse. Users, through editing such objects, can communicate with other individuals on mobile headsets. Knowing that the tangible controllers cause the burden to carry such addendum devices, the body-centric interaction techniques, such as hand gestures, get rid of such burdens. However, object editing with hand gestures is usually overlooked. Accordingly, we propose and implement a palm-based virtual embodiment for hand gestural model editing, namely 3DeformR. We employ three optimized hand gestures on bi-harmonic deformation algorithms that enable selecting and editing 3D models in fine granularity. Our evaluation with nine participants considers three interaction techniques (two-handed tangible controller (OMC), a naive implementation of hand gestures (SH), and 3DeformR. Two experimental tasks of planar and spherical objects imply that 3DeformR outperforms SH, in terms of task completion time (~51%) and required actions (~17%). Also, our participants with 3DeformR make significantly better performance than the commercial standard (OMC) - saved task time (~43%) and actions (~3%). Remarkably, the edited objects by 3DeformR show no discernible difference from those with tangible controllers characterised by accurate and responsive detection.
Explainability methods for machine learning systems for multimodal medical datasets

This paper contains the research proposal of Andrea M. Storås that was presented at the MMSys 2022 doctoral symposium. Machine learning models have the ability to solve medical tasks with a high level of performance, e.g., classifying medical videos and detecting anomalies using different sources of data. However, many of these models are highly complex and difficult to understand. Lack of interpretability can limit the use of machine learning systems in the medical domain. Explainable artificial intelligence provides explanations regarding the models and their predictions. In this PhD project, we develop machine learning models for automatic analysis of medical data and explain the results using established techniques from the field of explainable artificial intelligence. Current research indicate that there are still open issues to be solved in order for end users to understand multimedia systems powered by machine learning. Consequently, new explanation techniques will also be developed. Different types of medical data are applied in order to investigate the generalizability of the methods.
Visual privacy protection in mobile image recognition using protective perturbation

Deep neural networks (DNNs) have been widely adopted in mobile image recognition applications. Considering intellectual property and computation resources, the image recognition model is often deployed at the service provider end, which takes input images from the user's mobile device and accomplishes the recognition task. However, from the user's perspective, the input images could contain sensitive information that is subject to visual privacy concerns, and the user must protect the privacy while offloading them to the service provider. To address the visual privacy issue, we develop a protective perturbation generator at the user end, which adds perturbations to the input images to prevent privacy leakage. Meanwhile, the image recognition model still runs at the service provider end to recognize the protected images without the need of being re-trained. Our evaluations using the CIFAR-10 dataset and 8 image recognition models demonstrate effective visual privacy protection while maintaining high recognition accuracy. Also, the protective perturbation generator achieves premium timing performance suitable for real-time image recognition applications.
A virtual reality volumetric music video

Music videos are short films that integrate songs and imagery produced for artistic and promotional purposes. Modern music videos apply various media capture techniques and creative postproduction technologies to provide a myriad of stimulating and artistic approaches to audience entertainment and engagement for viewing across multiple devices. Within this domain, volumetric video capture technologies (Figure 2) have become an emerging means of recording and reproducing musical performances for new audiences to access via traditional 2D screens and emergent extended reality platforms, such as augmented and virtual reality. These 3D digital reproductions of musical performances are captured live and are enhanced to deliver cutting-edge audiovisual entertainment (Figure 1). However, the precise impact of volumetric video in music video entertainment is still in a state of flux.
Enhancing situational awareness with adaptive firefighting drones

High-rise fires are among the largest threats to safety in modern cities, and autonomous drones with multi-modal sensors can be employed to enhance situational awareness in such unfortunate disasters. In this paper, we study the fine-grained measurement selection problem for drones being dispatched to perform situation monitoring tasks in high-rise fires. Our problem considers multiple sensor/media types, classifier designs, and measurement locations, which were overlooked in prior waypoint scheduling studies. For concrete discussion, we adopt window openness as the target situation, while other situations can be readily supported by our solution as well. More specifically, we: (i) develop diverse window openness classifiers, (ii) mathematically formulate the fine-grained measurement selection problem and solve it using two algorithms, and (iii) create a photo-realistic simulator and an event-driven simulator to evaluate our algorithms. The evaluation results demonstrate that our proposed algorithms achieve higher classification accuracy (up to 50% improvement), deliver more feasible solutions (up to 100% improvement), and reduce energy consumption (up to 6.78 times reduction), compared to the current practices.
Unsupervised method for video action segmentation through spatio-temporal and positional-encoded embeddings

Action segmentation consists of temporally segmenting a video and labeling each segmented interval with a specific action label. In this work, we propose a novel action segmentation method that requires no prior video analysis and no annotated data. Our method involves extracting spatio-temporal features from videos using a pre-trained deep network. Data is then transformed using a positional encoder, and finally a clustering algorithm is applied, where each produced cluster presumably corresponds to a different single and distinguishable action. In experiments, we show that our method produces competitive results on the Breakfast and Inria Instructional Videos dataset benchmarks.
A QoE evaluation of procedural and example instruction formats for procedure training in augmented reality

Augmented reality (AR) has significant potential as a training platform. The pedagogical purpose of training is learning or transfer. Learning is the acquisition of an ability to perform a procedure as taught while transfer involves generalising that knowledge to similar procedures in the same domain. Quality of experience (QoE) concerns the fulfilment of the application, system or service user's pragmatic and hedonic needs and expectations. Learning or transfer fulfil the AR trainee's pragmatic needs. Training instructions can be presented in procedural, and example formats. Procedural instructions tell the trainee what to do while examples show the trainee how to do it. These two different instruction formats can influence learning, transfer, and hardware resource availability differently. The AR trainee's hedonic needs and expectations may be influenced by the impact of instruction format resource consumption on system performance. Efficient training efficacy is a design concern for mobile AR training applications. This work aims to inform AR training application design by evaluating the influence of procedural and example instruction formats on AR trainee QoE.