Miracle-dz/UniFAR-FTUs huggingface dataset:
From databases to artificial intelligence

This habilitation thesis synthesizes our research efforts in the area of relational databases and artificial intelligence. In particular, we present our work on various data matching problems, where we need to establish connections between different pieces of information, such that these correspondences reflect our human understanding of the relation among them. Entity resolution aims to identify entity references (person or company names, geographic locations) such that they refer to the same real word entity. We summarize here our work on this problem in the context of Web documents. Our methods rely on supervised machine learning techniques. We discuss our work on database schema matching. Our work addressed a specific setting of this problem, where we need to match a set of schemas based on a network of their pairwise interactions. Even if available schema matching tools can obtain a set of good quality attribute correspondences, if we would like to use them for data integration, we need to eliminate the remaining errors. This phase still requires the involvement of human experts. We model this post-matching phase and we propose new techniques to reduce the necessary human efforts and to guide the work of the experts. Our methods rely on (probabilistic) reasoning methods that exploit the relevant consistency constraints. Crowdsourcing is a model where one can request a service that is realized by human workers form from a large crowd of users, in exchange for a small payment. Crowdsourcing platforms enable human-in-the-loop algorithms and they were also used to construct large labelled datasets. A central issue in this setting is the quality of the obtained results, in particular in the context of knowledge-intensive crowdsourcing, where workers need specific skills to complete the requested task. Affecting the tasks to workers who do not have these skills could lead to poor quality results. We propose new methods which can improve the expected quality of the results through the use of simple forms of reasoning on human skills. We also discuss our perspectives for future research directions.
Effect of User Roles on the Process of Collaborative 2D Level Design on Large, High-resolution Displays

: This paper presents groupware to study group behavior while conducting a creative task on large, high-resolution displays. Moreover, we present the results of a between-subjects study. In the study, 12 groups with two participants each prototyped a 2D level on a 7m x 2.5m large, high-resolution display using tablet-PCs for interaction. Six groups underwent a condition where group members had equal roles and interaction possibilities. Another six groups worked in a condition where group members had different roles: level designer and 2D artist. The results revealed that in the different roles condition, the participants worked signiﬁcantly more tightly and created more assets. We could also detect some shortcomings for that conﬁguration. We discuss the gained insights regarding system conﬁguration, groupware interfaces, and groups behavior.
A Genetic Algorithm Optimising Control Point Placement for Edge Bundling

: This paper describes a novel approach of edge bundling that employs a genetic algorithm (GA) to optimise the placement of control points. Edge bundling is a useful technique to reduce visual clutter and a number of model-based edge bundling approaches have been introduced in the literature. However, these do not attempt to optimise aesthetic rules directly. Differently from them, our approach assumes that edge bundling is regarded as an optimisation problem for aesthetic rules. To solve this problem, we present an GA-based algorithm where gene representation defines control points of edges in order to allow flexibility and the fitness function is defined based on quantitative criteria for edge bundling. Experimental results using a visualisation of a Japanese airline map demonstrates the feasibility of our proposed method and its usability.
Image-based Material Editing for Making Reflective Objects Fluorescent

: Fluorescent materials give us a unique sense of quality such as self-luminous ones, because they absorb light with certain wavelengths and then emit light with longer wavelengths. The existing methods for image-based material editing make objects in an image specular, translucent, and transparent, but they do not address ﬂuorescent materials. In this paper, we propose a method for making reﬂective objects in a single input image ﬂuorescent by adding photorealistic ﬂuorescent components to the objects of interest. Speciﬁcally, we show that photometrically consistent ﬂuorescent components can approximately be represented by using the 3-band (RGB) spectral irradiance on the surface of a reﬂective object, and then compute the ﬂuorescent components on the basis of intrinsic image decomposition without explicitly estimating the object’s shape and the light sources illuminating it from the input image. We conducted a number of experiments using both synthetic and real images, and conﬁrmed that our proposed method is effective for making reﬂective objects ﬂuorescent.
Validation of a Low-cost Inertial Exercise Tracker

This work validates the application of a low-cost inertial tracking suit, for strength exercise monitoring. The procedure includes an offline processing for body-IMU calibration and an online tracking and identification of lower body motion. We proposed an optimal movement pattern for the body-IMU calibration method from our previous work. Here, in order to reproduce real extreme situations, the focus is on the movements with high acceleration. For such movements, an optimal orientation tracking approach is introduced, which requires no accelerometer measurements and it thus minimizes error due to outliers. The online tracking algorithm is based on an extended Kalman filter(EKF), which estimates the position of upper and lower legs, along with hip and knee joint angles. This method applies the estimated values in the calibration process i.e. joint axes and positions, as well as biomechanical constraints of lower body. Therefore it requires no aiding sensors such as magnetometer. The algorithm is evaluated using optical tracker for two types of exercises: squat and hip abd/adduction which resulted average root mean square error(RMSE) of 9cm. Additionally, this work presents a personalized exercise identification approach, where an online template matching algorithm is applied and optimised using zero velocity crossing(ZVC) for feature extraction. This results reducing the execution time to 93% and improving the accuracy up to 33%.
Detecting and Locating Boats using a PTZ Camera with Both Optical and Thermal Sensors

A harbor traffic monitoring system is necessary for most ports, yet current systems are often not able to detect and receive information from boats without transponders. In this paper we propose a computer vision based monitoring system utilizing the multi-modal properties of a PTZ (pan, tilt, zoom) camera with both an optical and thermal sensor in order to detect boats in different lighting and weather conditions. In both domains boats are detected using a YOLOv3 network pretrained on the COCO dataset and retrained using transfer-learning to images of boats in the test environment. The boats are then positioned on the water using ray-casting. The system is able to detect boats with an average precision of 95.53% and 96.82% in the optical and thermal domains, respectively. Furthermore, it is also able to detect boats in low optical lighting conditions, without being trained with data from such conditions, with an average precision of 15.05% and 46.05% in the optical and thermal domains, respectively. The position estimator, based on a single camera, is able to determine the position of the boats with a mean error of 18.58 meters and a standard deviation of 17.97 meters.
Adoption of Sparse 3D Textures for Voxel Cone Tracing in Real Time Global Illumination

: The enhancement of 3D scenes using indirect illumination brings increased realism. As indirect illumination is computationally expensive, signiﬁcant research effort has been made in lowering resource requirements while maintaining ﬁdelity. State-of-the-art approaches, such as voxel cone tracing, exploit the parallel nature of the GPU to achieve real-time solutions. However, such approaches require bespoke GPU code which is not tightly aligned to the graphics pipeline in hardware. This results in a reduced ability to leverage the latest dedicated GPU hardware graphics techniques. In this paper we present a solution that utilises GPU supported sparse 3D texture maps. In doing so we provide an engineered solution that is more integrated with the latest GPU hardware than existing approaches to indirect illumination. We demonstrate that our approach not only provides a more optimal solution, but will beneﬁt from the planned future enhancements of sparse 3D texture support expected in GPU development.
A Timeline Metaphor for Analyzing the Relationships between Musical Instruments and Musical Pieces

: Digitization projects make cultural heritage data sustainably available. However, while digital libraries may capture various aspects, relations across different sources often remain unobserved. In our project, musicol-ogists aimed to relate musical instruments with historical performances of musical pieces, both contained in different sources. We deﬁned a similarity measure taking instrumentation, temporal as well as geospatial metadata into account, with which we were able to hypothesize potential relations. We propose a novel time-line design that offers a speciﬁc semantic zoom metaphor enabling the collaborating musicologists to observe and evaluate the results of our similarity analysis. The value of our system for research in musicology is documented in three case studies.
User Time Spent between Persuasiveness and Usability of Social Networking Mobile Applications: Patterns of Influence

: Using social media is one of the most common activities for mobile users. Moreover, it is a time-consuming activity that can lead to addiction. Some gaps in HCI (Human Computer Interaction) ergonomics theory gave rise to this addiction. These gaps lie in an overexploitation of the usability and/or persuasion criteria that designers and/or developers use according to their needs when applying influence strategies to affect users’ engagement. Although these strategies are widely applied in online social networks, they are not well identified and their application levels are still lacking. This paper seeks to establish and validate these influence strategies. We proposed five (05) patterns of influence in online social networks that have a significant impact on Users’ Time Spent (UTS) grouping the different usability criteria and persuasion strategies. Then, we conducted a classification study of those criteria / strategies, using Hybrid Card Sort method carried out by fifteen (15) eligible experts. Experts were asked to group those criteria / strategies into a set of patterns based on our predetermined (with the option to create their own patterns). The results analysis validates our five (05) proposed patterns paving the way to outline their application borderline thereafter.
Anomaly Event Detection based on People Trajectories for Surveillance Videos

: In this work, we propose a novel approach to detect anomalous events in videos based on people movements, which are represented through time as trajectories. Given a video scenario, we collect trajectories of normal behavior using people pose estimation techniques and employ a multi-tracking data association heuristic to smooth trajectories. We propose two distinct approaches to describe the trajectories, one based on a Convolu-tional Neural Network and second based on a Recurrent Neural Network. We use these models to describe all trajectories where anomalies are those that differ much from normal trajectories. Experimental results show that our model is comparable with state-of-art methods and also validates the idea of using trajectories as a resource to compute one type of useful information to understand people behavior; in this case, the existence of rare trajectories.
S3D-R2R: An Automatic Stereoscopic 3D Image Recomposition to Retargeting Method with Depth Modification

: Stereoscopic image adaptation to the target display devices while minimizing the distortion of signiﬁcant features and stereoscopic properties is a challenging problem. Conventional methods either fail to preserve the image context or unable to improve the image aesthetics with improved depth perception in the retargeted images. In this paper, we present an automatic warping-based stereoscopic 3D image recomposition to retar-geting method , shortly S3D-R2R that improves the stereo image composition in the retargeting results. Our S3D-R2R method resizes both the left and right stereo image pair using a global optimization algorithm that minimizes a set of aesthetic quality errors. These errors are formulated based on the selected photographic composition rules and modify the depth perception. To improve the depth perception of the stereo image pair, the disparity consistency has been modiﬁed within the comfort disparity range. Experimental results show that our automatic method changes the position of the salient object in the target image scale and improves the depth perception within the comfort depth range. Empirical user studies indicate that our retargeting results receive more attention than state-of-the-art methods.
Automatic Classification of Cervical Cell Patches based on Non-geometric Characteristics

: This work presents a proposal for an efﬁcient classiﬁcation of cervical cells based on non-geometric characteristics extracted from nuclear regions of interested. This approach is based on the hypothesis that the nuclei store much of the information about the lesions in addition to their areas being more visible even with a high level of celular overlap, a common fact in the Pap smears images. Classiﬁcation systems were used in two and three classes for a set of real images of the cervix from a supervised learning method. The results demonstrate high classiﬁcation performance and high efﬁciency for applicability in realistic environments, both computational and biological.
Geovisualization: Multidimensional Exploration of the Territory

: The purpose of this position paper is to emphasize the remaining challenges for geovisualization in an evo-lutive context of data, users and spatio-temporal problems to solve in an interdisciplinary approach. Geovi-sualization is the visualization of spatio-temporal data, phenomena and dynamics on earth, based on the user interaction with heterogeneous data, and their capacities of perception and cognition. This implies to bring closer together knowledge, concepts and models from related scientiﬁc visualization domains, for a better understanding, interpretation and analysis of spatio-temporal phenomena on earth. We currently face and cross several types of complexities, regarding spaces, data, models and tools. Our position here, based on past and on-going works, as ﬁrst proofs of concept, is to model a multidimensional exploration of the territory, because integrating explorations of uses, styles, interaction and immersion capacities, until various ’points of view’ on the represented spatio-temporal phenomenon.
Trajectory Extraction and Deep Features for Classification of Liquid-gas Flow under the Context of Forced Oscillation

: Computer vision and deep learning techniques are increasingly applied to analyze experimental processes in engineering domains. In this paper, we propose a new dataset of liquid-gas ﬂow videos captured from a mechanical model simulating a cooling gallery of an automobile engine, through forced oscillations. The analysis of this dataset is of interest for ﬂuid-mechanic ﬁeld to validate the simulation environment. From computer vision point of view, it provides a new dynamic texture dataset with challenging tasks since liquid and gas keep changing constantly and the form of liquid-gas ﬂow is closely related to the external environment. In particular predicting the rotation velocity of the engine corresponding to liquid-gas movements is a ﬁrst step before precise analysis of ﬂow patterns and of their trajectories. The paper also provides an experimental analysis showing that such rotation velocity can be hard to predict accurately. It could be achieved using deep learning approaches but not with state-of-the-art method dedicated to trajectory analysis. We show also that a preprocessing step with difference of Gaussian (DoG) over multiple scales as input of deep neural networks is mandatory to obtain satisfying results, up to 81.39% on the test set. This study opens an exploratory ﬁeld for complex tasks on dynamic texture analysis such as trajectory analysis of heterogeneous masses.
Matching-aware Shape Simplification

Current research has shown significant interest in spatio-temporal data. The acquisition of spatio-temporal data usually begins with the segmentation of the objects of interest from raw data, which are then simplified and represented as polygons (contours). However, the simplification is usually performed individually, i.e., one polygon at a time, without considering additional information that can be inferred by looking at the correspondences between the polygons obtained from consecutive snapshots. This can reduce the quality of polygon matching, as the simplification algorithm may choose to remove vertices that would be relevant for the matching and maintain other less relevant ones. This causes undesired situations like unmatched vertices and multiple matched vertices. This paper presents a new methodology for polygon simplification that operates on pairs of shapes. The aim is to reduce the occurrence of unmatched and multiple matched vertices, while maintaining relevant vertices for image representation. We evaluated our method on synthetic and real world data and performed an extensive comparative study with two well-known simplification algorithms. The results show that our method outperforms current simplification algorithms, as it reduces the amount of unmatched vertexes and of vertexes with multiple matches.
Transparent Parallelization of Enrichment Operations in Geometric Modeling

This paper presents an approach to automatically and transparently parallelize algorithms to build 2D or 3D virtual objects in geometric modeling: In particular, we show that subdivision and Iterated Function System constructions can be parallelized, without any explicit parallelization study by their developer. These operations are described in the framework Jerboa, where each operation is expressed as a graph transformation and objects are topologically described using generalized maps. All transformations are handled by a generic engine that can handle structure changes in parallel. The obtained results allow any designer of virtual environments to rely on modern multi-core and multi-processor architectures to get faster constructions of complex objects without any skills on parallelism.
A Model based Toolchain for the Cosimulation of Cyber-physical Systems with FMI

: Smart Grids are cyber-physical systems that interface power grids with information and communication technologies in order to monitor them, automate decision making and balance production and consumption. Cosimulation with the Functional Mock-up Interface standard allows the explo-ration of the behavior of such complex systems by coordinating simulation units that correspond to the grid part, the communication network and the information system. However, FMI has limitations when it comes to cyber-physical system simulation, particularly because discrete-event signals exchanged by cyber components are not well supported. In addition, industrial projects involve several teams with diﬀerent skills and methods that work in parallel to produce all the models required by the simulation, which increases the risk of inconsistency between models. This article presents a way to exchange discrete-event signals between FMI artifacts, which complies with the current 2.0 version of the standard. We developed a DSL and a model-based toolchain to generate the artifacts that are necessary to run the cosimulation of the whole system, and to detect potential inconsistencies between models. The approach is illustrated by the use case of an islanded grid implementing diesel and renewable sources, battery storage and intelligent control of the production.
Multi-stream Deep Networks for Vehicle Make and Model Recognition

Vehicle recognition generally aims to classify vehicles based on make, model and year of manufacture. It is a particularly hard problem due to the large number of classes and small inter-class variations. To handle this problem recent state of the art methods use Convolutional Neural Network (CNN). These methods have however several limitations since they extract unstructured vehicle features used for the recognition task. In this paper, we propose more structured feature extraction method by leveraging robust multi-stream deep networks architecture. We employ a novel dynamic combination technique to aggregate different vehicle part features with the entire image. This allows combining global representation with local features. Our system which has been evaluated on publicly available datasets is able to learn highly discriminant representation and achieves state-of-the-art result.
A Hybrid Approach for Segmenting and Fitting Solid Primitives to 3D Point Clouds

: The segmentation and ﬁtting of solid primitives to 3D point clouds is a complex task. Existing systems are restricted either in the number of input points or the supported primitive types. This paper proposes a hybrid pipeline that is able to reconstruct spheres, bounded cylinders and rectangular cuboids on large point sets. It uses a combination of deep learning and classical RANSAC for primitive ﬁtting, a DBSCAN-based clustering scheme for increased stability and a specialized Genetic Algorithm for robust cuboid extraction. In a detailed evaluation, its performance metrics are discussed and resulting solid primitive sets are visualized. The paper concludes with a discussion of the approach’s limitations.
3D Convolutional Neural Network for Falling Detection using Only Depth Information

: Nowadays, one of the major challenges global society is facing is population aging, which involves an increment of the medical expenses. Since falls are the major cause of injuries for elderly people, the need of a low-cost falling detector has increased rapidly over the years. In this context, we propose a fall-detection system based on 3D Convolutional Neural Networks (3D-CNN). Due to the fact that the system only uses depth information obtained by a RGB-D sensor placed in a overhead position to avoid occlusions, it results in a less invasive and intrusive fall-detection method for users than systems based on wearables. In addition, depth information preserves the privacy of people since they cannot be identiﬁed from this information. The 3D-CNN obtains spatial and temporal features from depth data, which allows to classify users’ actions and detect when a fall appears. Since there are no other available datasets for action recognition using only depth data from a top-view camera, the authors have recorded and labeled the GOTPD3, that has been made available to the scientiﬁc community. Thus, training and evaluation of the network has been carried out within the GOTPD3 dataset, and the achieved results validate the proposal.