Reviewerly/exHarmony huggingface dataset:
Creating a Culture of Innovation

In this chapter, we describe the design and implementation of a Pedagogical Innovation Center (PIC) at the Polytechnic of Porto. The COVID-19 pandemic disrupted our day to day lives, our businesses, the world trade and movements. Education was not spared. In fact, it was one of the sectors most heavily affected by COVID-19 pandemic. Teachers were forced, from night to day, to adjust a purely face-to-face teaching style, to a 100\% online set. This is known as emergency remote teaching. Several difficulties have arisen both for teachers and students. The first had to structure all their teaching materials from scratch, had to design and apply new assessment methods, and struggled to get their students' motivation. On their side, the students lacked engagement, social interaction with peers and teachers, the ability to have a more autonomous learning style.
A STUDY OF DATA AUGMENTATION AND ACCURACY IMPROVEMENT IN MACHINE TRANSLATION FOR VIETNAMESE SIGN LANGUAGE

Sign languages are independent languages of deaf communities. The translation from normal languages (i.e., Vietnamese Language - VL) as long as other sign languages to Vietnamese sign language (VSL) is a meaningful task that breaks down communication barriers and improves the quality of life for the deaf community. In this paper, we experimented with and proposed several methods for building and improving models for the VL to VSL translation task. We presented a data augmentation method to improve the performance of our neural machine translation models. Using an initial dataset of 10k bilingual sentence pairs, we were able to obtain a new dataset of 60k sentence pairs with a perplexity score no more than 1.5 times that of the original dataset. Experiments on the original dataset showed that rule-based models achieved the highest BLEU score of 68.02 among the translation models. However, with the augmented dataset, the Transformer model achieved the best performance with a BLEU score of 89.23, which is significantly better than that of other conventional approach methods.
Homography based Player Identification in Live Sports

Modern live sports broadcasts display a wide variety of graphic visualizations identifying key players in a particular play. Traditionally, these graphics are created with extensive manual annotation for post-match analysis and take a significant amount of time to be produced. To create such visualizations in near real-time, automatic on-screen player identification and localization is essential. However, it is a challenging vision problem, especially for sports such as American football where the players wear elaborate protective equipment. In this work, we propose a novel approach which uses sensor data streams captured by wear-ables to automatically identify and locate on-screen players with low latency and high accuracy. The approach estimates a field registration homography using on-field player positions from RFID sensors, which is then used to identify and locate individual players on-screen. Experiments using American football data show that the method outperforms a deep learning based state-of-the-art(SOTA) vision-only field registration model both in terms of accuracy of the homography and also success rate of correct homography computation. On a dataset of over 150 replay clips, the proposed method correctly estimated the homography for approximately 25% additional clips as compared to the SOTA method. We demonstrate the efficacy of our method by applying it to the problem of rendering visualizations around key players within a few minutes of the live play. The player identification accuracy for these key players was over 96% across all clips, with an end-to-end latency of less than 1 minute.
A Data-Driven Approach based on Dynamic Mode Decomposition for Efficient Encoding of Dynamic Light Fields

Dynamic light fields provide a richer, more realistic 3D representation of a moving scene. However, this leads to higher data rates since excess storage and transmission requirements are needed. We propose a novel approach to efficiently represent and encode dynamic light field data for display applications based on dynamic mode decomposition (DMD). Acquired images are firstly obtained through optimized coded aperture patterns for each temporal frame/camera viewpoint of a dynamic light field. The underlying spatial, angular, and temporal correlations are effectively exploited by a data-driven DMD on these acquired images arranged as time snapshots. Next, High Efficiency Video Coding (HEVC) removes redundancies in light field data, including intra-frame and inter-frame redundancies, while maintaining high reconstruction quality. The proposed scheme is the first of its kind to treat light field videos as mathematical dynamical systems, leverage on dynamic modes of acquired images, and gain flexible coding at various bitrates. Experimental results demonstrate our scheme’s superior compression efficiency and bitrate savings compared to the direct encoding of acquired images using HEVC codec.
Human Spine Motion Capture using Perforated Kinesiology Tape

In this work, we present a marker-based multi-view spine tracking method that is specifically adjusted to the requirements for movements in sports. A maximal focus is on the accurate detection of markers and fast usage of the system. For this task, we take advantage of the prior knowledge of the arrangement of dots in perforated kinesiology tape. We detect the tape and its dots using a Mask R-CNN and a blob detector. Here, we can focus on detection only while skipping any image-based feature encoding or matching. We conduct a reasoning in 3D by a linear program and Markov random fields, in which the structure of the kinesiology tape is modeled and the shape of the spine is optimized. In comparison to state-of-the-art systems, we demonstrate that our system achieves high precision and marker density, is robust against occlusions, and capable of capturing fast movements.
Are Local Features All You Need for Cross-Domain Visual Place Recognition?

Visual Place Recognition is a task that aims to predict the coordinates of an image (called query) based solely on visual clues. Most commonly, a retrieval approach is adopted, where the query is matched to the most similar images from a large database of geotagged photos, using learned global descriptors. Despite recent advances, recognizing the same place when the query comes from a significantly different distribution is still a major hurdle for state of the art retrieval methods. Examples are heavy illumination changes (e.g. night-time images) or substantial occlusions (e.g. transient objects) . In this work we explore whether re-ranking methods based on spatial verification can tackle these challenges, following the intuition that local descriptors are inherently more robust than global features to domain shifts. To this end, we provide a new, comprehensive benchmark on current state of the art models. We also introduce two new demanding datasets with night and occluded queries, to be matched against a citywide database. Code and datasets are available at https://github.com/gbarbarani/re-ranking-for-VPR.
Exposing GAN-Generated Profile Photos from Compact Embeddings

Generative adversarial networks (GANs) have been used to create remarkably realistic images of people. More recently, diffusion-based techniques have taken image synthesis to the next level. From only a text prompt, these techniques can synthesize any image seemingly limited only by our imagination. Along with the many clever and creative use cases, synthetically-generated faces are being used to create more convincing fake social-media profiles. We describe two related techniques that learn low-dimensional (128-D) embeddings of GAN-generated faces. We show that these embeddings capture common facial structures found in these synthetically-generated faces that are uncommon in real profile photos. These low-dimensional models, trained on a relatively small data set, achieve higher classification performance than larger and more complex state-of-the-art classifiers.
Improving Data-Efficient Fossil Segmentation via Model Editing

Most computer vision research focuses on datasets containing thousands of images of commonplace objects. However, many high-impact datasets, such as those in medicine and the geosciences, contain fine-grain objects that require domain-expert knowledge to recognize and are time-consuming to collect and annotate. As a result, these datasets contain few labeled images, and current machine vision models cannot train intensively on them. Originally introduced to correct large-language models, model-editing techniques in machine learning have been shown to improve model performance using only small amounts of data and additional training. Using a Mask R-CNN to segment ancient reef fossils in rock sample images, we present a two-part paradigm to improve fossil segmentation with few labeled images: we first identify model weaknesses using image perturbations and then mitigate those weaknesses using model editing.Specifically, we apply domain-informed image perturbations to expose the Mask R-CNN’s inability to distinguish between different classes of fossils and its inconsistency in segmenting fossils with different textures. To address these shortcomings, we extend an existing model-editing method for correcting systematic mistakes in image classification to image segmentation with no additional labeled data needed and show its effectiveness in decreasing confusion between different kinds of fossils. We also highlight the best settings for model editing in our situation: making a single edit using all relevant pixels in one image (vs. using multiple images, multiple edits, or fewer pixels). Though we focus on fossil segmentation, our approach may be useful in other similar fine-grain segmentation problems where data is limited.
Federated Learning in Non-IID Settings Aided by Differentially Private Synthetic Data

Federated learning (FL) is a privacy-promoting framework that enables potentially large number of clients to collaboratively train machine learning models. In an FL system, a server coordinates the collaboration by collecting and aggregating clients' model updates while the clients' data remains local and private. A major challenge in federated learning arises when the local data is non-iid - the setting in which performance of the learned global model may deteriorate significantly compared to the scenario where the data is identically distributed across the clients. In this paper we propose FedDPMS (Federated Differentially Private Means Sharing), an FL algorithm in which clients augment local datasets with data synthesized using differentially private information collected and communicated by a trusted server. In particular, the server matches the pairs of clients having complementary local datasets and facilitates differentially-private sharing of the means of latent data representations; the clients then deploy variational auto-encoders to enrich their datasets and thus ameliorate the effects of non-iid data distribution. Our experiments on deep image classification tasks demonstrate that FedDPMS outperforms competing state-of-the-art FL methods specifically developed to address the challenge of federated learning on non-iid data.
High-Perceptual Quality JPEG Decoding via Posterior Sampling

JPEG is arguably the most popular image coding format, achieving high compression ratios via lossy quantization that may create visual artifacts degradation. Numerous attempts to remove these artifacts were conceived over the years, and common to most of these is the use of deterministic post-processing algorithms that optimize some distortion measure (e.g., PSNR, SSIM). In this paper we propose a different paradigm for JPEG artifact correction: Our method is stochastic, and the objective we target is high perceptual quality – striving to obtain sharp, detailed and visually pleasing reconstructed images, while being consistent with the compressed input. These goals are achieved by training a stochastic conditional generator (conditioned on the compressed input), accompanied by a theoretically well-founded loss term, resulting in a sampler from the posterior distribution. Our solution offers a diverse set of plausible and fast reconstructions for a given input with perfect consistency. We demonstrate our scheme’s unique properties and its superiority to a variety of alternative methods on the FFHQ and ImageNet datasets.
Three Recipes for Better 3D Pseudo-GTs of 3D Human Mesh Estimation in the Wild

Recovering 3D human mesh in the wild is greatly challenging as in-the-wild (ITW) datasets provide only 2D pose ground truths (GTs). Recently, 3D pseudo-GTs have been widely used to train 3D human mesh estimation networks as the 3D pseudo-GTs enable 3D mesh supervision when training the networks on ITW datasets. However, despite the great potential of the 3D pseudo-GTs, there has been no extensive analysis that investigates which factors are important to make more beneficial 3D pseudo-GTs. In this paper, we provide three recipes to obtain highly beneficial 3D pseudo-GTs of ITW datasets. The main challenge is that only 2D-based weak supervision is allowed when obtaining the 3D pseudo-GTs. Each of our three recipes addresses the challenge in each aspect: depth ambiguity, sub-optimality of weak supervision, and implausible articulation. Experimental results show that simply re-training state-of-the-art networks with our new 3D pseudo-GTs elevates their performance to the next level without bells and whistles. The 3D pseudo-GT is publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
A Light-Weight Human Eye Fixation Solution for Smartphone Applications

A wide range of human eye fixation prediction algorithms have been presented in the research with the advent of deep learning. However, to generate better prediction outcomes, these methods are becoming increasingly complicated. In this study, we present a lightweight human eye fixation prediction network that is based on a low-complexity representation learning network and can handle a variety of real-world data. The method includes a simplified multilevel feature extraction network with an emphasis on channel and spatial attention mechanism. We investigate the effectiveness of the present technique in predicting eye fixation maps on a collection of challenging images from the SALICON and MIT1003 datasets. A comprehensive qualitative and quantitative evaluation revealed that the network could learn and capture spatial and semantic information in a scene effectively, resulting in a higher hit rate and fewer false positives in comparison with the competing solutions. The approach is implemented on Samsung Galaxy S23 with SnapDragon-SM8550 mobile platform given its short inference time of 1.4ms and low model complexity.
Hard-negative Sampling with Cascaded Fine-Tuning Network to Boost Flare Removal Performance in the Nighttime Images

When light passes through a camera lens, it creates a residue called "flare" due to the interaction between foreign substances on the lens surface and internal glasses. At night, images can be distorted by flare due to multiple light sources, and research has been conducted using neural networks to remove the flare and solve this problem. However, to our knowledge, research on this approach has only recently begun, and the results are still limited, with only a few models available for use. Further research is needed to determine if the existing models provide optimal results. As part of the mentioned research, we propose a cascaded neural network structure as a means of fine-tuning earlier models to improve their performance. We optimize the performance of the proposed model by constructing triplets using the outputs of two identical neural networks and applying contrastive learning. To demonstrate the superiority of the proposed method, we quantitatively evaluated it by measuring PSNR and SSIM. We also visually compared the differences in image details after removing the flare. Experimental results confirmed that the images reconstructed by the proposed model were superior in terms of PSNR and SSIM in streak regions, compared to the results generated by the reference model.
Universal Watermark Vaccine: Universal Adversarial Perturbations for Watermark Protection

As computing ability continues to rapidly develop, neural networks have found widespread use in various fields. However, in the realm of visible watermarking for image copyright protection, neural networks have made image protection through watermarking less effective. Some research has even shown that watermarks can be removed without damaging to the original image, posing a significant threat to digital copyright protection. In response, the community has introduced adversarial perturbations for watermark protection, but these are sample-specific and time-consuming in real-world scenarios. To address this issue, we propose a new universal adversarial perturbation for watermark removal networks that offers two options. The first option involves adding perturbations to the entire host image, bringing the output of the watermark removal network closer to the original image and providing protection. The second option involves adding perturbations only to the watermark position, reducing the impact of the perturbation on the image and enhancing stealthiness. Our experiments demonstrate that our method effectively resists watermark removal networks and has good generalizability across different images.
Neural Transformation Network to Generate Diverse Views for Contrastive Learning

Recent unsupervised representation learning methods rely heavily on various transformations to generate distinctive views of given samples. Transformations for these views are generally defined manually, requiring significant human effort to design detailed configurations and validate practical efficacy. Furthermore, the diversity of these views is quite limited in scope causing the network to be invariant to only a small set of data transformations. To address these problems, we introduce a neural transformation network that learns to generate diverse views. Our proposed framework consists of an encoder-decoder network architecture that encodes semantic information and then randomly stylizes it with style amplification. However, such generative processes tend to cause degradation compared to the original images, which can harm the quality of the learned representation. To remedy this issue and generate more diverse styles, we use a linear augmentation between the generated view and the original image. Finally, we apply geometric transformations to aid in contrastive learning of representations. We evaluate the learned representations on various downstream vision tasks. Results show highly competitive recognition performance compared to the state-of-the-art methods that use learned views or hand-crafted views for representation learning.
Zero-shot Object Classification with Large-scale Knowledge Graph

Zero-shot learning is research for predicting unseen categories, and can solve problems such as dealing with unseen categories that were not anticipated at the time of training and the lack of labeled datasets. One of the methods for zero-shot object classification is using a knowledge graph, which is a set of explicit knowledge. Since recognition is limited to the categories contained in the knowledge graph and the relationships among categories are expected to be quantitatively and qualitatively richer depending on the graph size, it is desirable to handle a large-scale knowledge graph that contains as many categories as possible. We use a knowledge graph that contains about seven times as many categories as the knowledge graphs used mainly in existing research to enable classification of a larger number of categories and to achieve more accurate recognition. When using large-scale knowledge graph, it is expected that the number of noisy nodes and edges will increase. Therefore we propose a method to extract useful information from entire graph by positional relationships between categories and the types of edges in the knowledge graph. We classify images that were unclassifiable in existing research and show that the proposed data extraction method improves performance compared to using entire graph.
Sparse Multimodal Vision Transformer for Weakly Supervised Semantic Segmentation

Vision Transformers have proven their versatility and utility for complex computer vision tasks, such as land cover segmentation in remote sensing applications. While performing on par or even outperforming other methods like Convolutional Neural Networks (CNNs), Transformers tend to require even larger datasets with fine-grained annotations (e.g., pixel-level labels for land cover segmentation). To overcome this limitation, we propose a weakly-supervised vision Transformer that leverages image-level labels to learn a semantic segmentation task to reduce the human annotation load. We achieve this by slightly modifying the architecture of the vision Transformer through the use of gating units in each attention head to enforce sparsity during training and thereby retaining only the most meaningful heads. This allows us to directly infer pixel-level labels from image-level labels by post-processing the un-pruned attention heads of the model and refining our predictions by iteratively training a segmentation model with high fidelity. Training and evaluation on the DFC2020 dataset show that our method <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> not only generates high-quality segmentation masks using image-level labels, but also performs on par with fully-supervised training relying on pixel-level labels. Finally, our results show that our method is able to perform weakly-supervised semantic segmentation even on small-scale datasets.
MIPI 2023 Challenge on RGBW Fusion: Methods and Results

Developing and integrating advanced image sensors with novel algorithms in camera systems are prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for an in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). With the success of the 1st MIPI Workshop@ECCV 2022, we introduce the second MIPI challenge, including four tracks focusing on novel image sensors and imaging algorithms. This paper summarizes and reviews the RGBW Joint Fusion and Denoise track on MIPI 2023. In total, 69 participants were successfully registered, and 4 teams submitted results in the final testing phase. The final results are evaluated using objective metrics, including PSNR, SSIM, LPIPS, and KLD. A detailed description of the models developed in this challenge is provided in this paper. More details of this challenge and the link to the dataset can be found at https://mipi-challenge.org/MIPI2023/.