Reviewerly/exHarmony huggingface dataset:
Digital forensic analysis of the private mode of browsers on Android

The smartphone has become an essential electronic device in our daily lives. We carry our most precious and important data on it, from family videos of the last few years to credit card information so that we can pay with our phones. In addition, in recent years, mobile devices have become the preferred device for surfing the web, already representing more than 50% of Internet traffic. As one of the devices we spend the most time with throughout the day, it is not surprising that we are increasingly demanding a higher level of privacy. One of the measures introduced to help us protect our data by isolating certain activities on the Internet is the private mode integrated in most modern browsers. Of course, this feature is not new, and has been available on desktop platforms for more than a decade. Reviewing the literature, one can find several studies that test the correct functioning of the private mode on the desktop. However, the number of studies conducted on mobile devices is incredibly small. And not only is it small, but also most of them perform the tests using various emulators or virtual machines running obsolete versions of Android. Therefore, in this paper we apply the methodology we presented in a previous work to Google Chrome, Brave, Mozilla Firefox, and Tor Browser running on a tablet with Android 13 and on two virtual devices created with Android Emulator. The results confirm that these browsers do not store information about the browsing performed in private mode in the file system. However, the analysis of the volatile memory made it possible to recover the username and password used to log in to a website or the keywords typed in a search engine, even after the devices had been rebooted.
High-Speed Optical Convolutional Neural Network Accelerator with 100 Gbaud EO-polymer/Si Hybrid Optical Modulator

Convolutional neural networks are a powerful category of artificial neural networks that can extract hierarchical features of biological data to significantly reduce parametric complexity and increase prediction accuracy. In this paper, we propose high-speed optical convolutional neural network accelerator with 100Gbaud EO-polymer/Si hybrid optical modulator and demonstrated proof of concept both in simulation and experiment.
On-chip Training Silicon Photonic Circuits for Novel Classification Computing

In this invited talk, we review our work on on-chip training silicon photonic circuits for novel classification computing based on support vector machine like projection-based principle and its recent progress.
A Reconfigurable Optical Network for Distributed Deep Learning

We propose a reconfigurable optical network to accelerate distributed deep learning. The optical network reconfigures the topology to adapt different all-reduce algorithms and models. We demonstrate the performance improvement with a 32-node prototype.
Where are they looking in the 3D space?

We propose a novel depth-aware joint attention target estimation framework that estimates the attention target in 3D space. Our goal is to mimic human’s ability to understand where each person is looking in their proximity. In this work, we tackle the previously unexplored problem of utilising a depth prior along with a 3D joint FOV probability map to estimate the joint attention target of people in the scene. We leverage the insight that besides the 2D image content, strong gaze-related constraints exist in the depth order of the scene and different subject-specific attributes. Extensive experiments show that our method outperforms favourably against existing joint attention target estimation methods on the VideoCoAtt benchmark dataset. Despite the proposed framework being designed for joint attention target estimation, we show that it outperforms single attention target estimation methods on both the GazeFollow image and the VideoAttentionTarget video benchmark datasets.
Federated Learning with Unsourced Random Access

A large number of new applications are emerging in the future sixth-generation (6G) communication systems. Federated learning (FL) enables massive user equipments (UEs), such as mobile phones and Internet of Things (IoT) devices, to cooperatively learn a shared model for prediction in various applications, while keeping the training data local. However, in practical scenarios, there are still some problems in deploying FL systems, including serving a large number of active UEs, longtime delay, and the risk of UEs’ privacy leakage. To tackle these issues, we introduce unsourced random access (URA) into the FL systems. URA can support massive connectivity and its unsourced property can protect the UEs’ identity privacy. Moreover, considering the trade-off between communication and computation performance and the various importance of different UEs’ local models in training epochs, two importance metrics are designed. The UEs can decide their own active probability according to the metrics among the communication rounds, which avoids the additional cost of being scheduled by the base station (BS) and maximums the use of the limited communication resources to ensure UEs with higher priority can upload trained models, thus improving the training efficiency. Simulation results verify the remarkable communication and computation performance of the proposed schemes.
Improving language-supervised object detection with linguistic structure analysis

Language-supervised object detection typically uses descriptive captions from human-annotated datasets. However, in-the-wild captions take on wider styles of language. We analyze one particular ubiquitous form of language: narrative. We study the differences in linguistic structure and visual-text alignment in narrative and descriptive captions and find we can classify descriptive and narrative style captions using linguistic features such as part of speech, rhetoric structure theory, and multimodal discourse. Then, we use this to select captions from which to extract image-level labels as supervision for weakly supervised object detection. We also improve the quality of extracted labels by filtering based on proximity to verb types for both descriptive and narrative captions.
FF-Former: Swin Fourier Transformer for Nighttime Flare Removal

In the process of removing nighttime flare, it is crucial to have a large receptive field due to the fact that flare can occupy a substantial portion of an image, even potentially the entire image. However, the conventional window-based Transformer approaches restrict the receptive field within the window, limiting its ability to capture global features. And the flare can cause the dark regions to become brighter and result in a loss of contrast and alteration of the frequency characteristics of the image. To address these challenges, we introduce FF-Former, which is based on Fast Fourier Convolution (FFC) and is designed to extract global frequency features for enhancing nighttime flare removal. To achieve this, we incorporate a Spatial Frequency Block (SFB) after the Swin Transformer, which forms the Swin Fourier Transformer Block (SFTB). This configuration enables the establishment of long dependencies and the extraction of global features. Unlike the traditional Transformer, which relies on global self-attention, the SFB module only performs convolution computation, making it both effective and efficient. Additionally, during the training phase, we optimize the loss function to preserve the light source points after nighttime flare removal. Experimental results on both real-world and synthetic benchmarks demonstrate that the proposed FF-Former significantly improves the performance of nighttime flare removal.
Device-Edge Digital Semantic Communication with Trained Non-Linear Quantization

Powered by deep learning, semantic communication is an intelligent communication paradigm, aiming to transmit useful information in the semantic domain. In most existing work, robust semantic features can be learned against wireless channel degradation, and directly transmitted in an analog fashion. However, analog semantic communication raises various challenges to the existing system from hardware/protocol to encryption issues. In this paper, we propose a novel non-linear quantization module to efficiently quantify semantic features. A sparse scaling vector is further incorporated to reduce the dimension of transmitted semantic features. Experimental results demonstrate that the proposed nonlinear quantization achieves better performance than the linear quantization method, and the performance of the digital system achieves better performance.
One-shot Learning for Channel Estimation in Massive MIMO Systems

In conventional supervised deep learning based channel estimation algorithms, a large number of training samples are required for offline training. However, in practical communication systems, it is difficult to obtain channel samples for every signal-to-noise ratio (SNR). Furthermore, the generalization ability of these deep neural networks (DNN) is typically poor. In this work, we propose a one-shot self-supervised learning framework for channel estimation in multi-input multi-output (MIMO) systems. The required number of samples for offline training is small and our approach can be directly deployed to adapt to variable channels. Our framework consists of a traditional channel estimation module and a denoising module. The denoising module is designed based on the one-shot learning method Self2Self and employs Bernoulli sampling to generate training labels. Besides,we further utilize a blind spot strategy and dropout technique to avoid overfitting. Simulation results show that the performance of the proposed one-shot self-supervised learning method is very close to the supervised learning approach while obtaining improved generalization ability for different channel environments.
Deduplication of Textual Data by NLP Approaches

With the increasing amount of digital data, data deduplication has become an increasingly popular method for reducing data in large-scale storage systems. Generalized deduplication is an alternative technique for reducing the cost of data storage by identifying similar data chunks. This paper proposes TL-GD, a method for improving cloud storage efficiency using generalized deduplication focusing on textual datasets. The core concept of this study is to develop an efficient deduplication system that combines an alternative technique for splitting data into smaller pieces and a new approach for transforming data pieces into bases and deviations. The performance of the system has been validated using two real-world datasets. We also compare the results to state-of-the-art deduplication methods. Our evaluation results show that TL-GD achieves nearly 67% lossless compression for textual navigation instructions datasets, which is a 25% improvement on average compared to existing deduplication techniques.
Benchmarking Robustness to Text-Guided Corruptions

This study investigates the robustness of image classifiers to text-guided corruptions. We utilize diffusion models to edit images to different domains. Unlike other works that use synthetic or hand-picked data for benchmarking, we use diffusion models as they are generative models capable of learning to edit images while preserving their semantic content. Thus, the corruptions will be more realistic and the comparison will be more informative. Also, there is no need for manual labeling and we can create large-scale benchmarks with less effort. We define a prompt hierarchy based on the original ImageNet hierarchy to apply edits in different domains. As well as introducing a new benchmark we try to investigate the robustness of different vision models. The results of this study demonstrate that the performance of image classifiers decreases significantly in different language-based corruptions and edit domains. We also observe that convolutional models are more robust than transformer architectures. Additionally, we see that common data augmentation techniques can improve the performance on both the original data and the edited images. The findings of this research can help improve the design of image classifiers and contribute to the development of more robust machine learning systems. The code for generating the benchmark will be made available on-line upon publication.
FedATM: Adaptive Trimmed Mean based Federated Learning against Model Poisoning Attacks

Federated learning (FL) has received explosive research attention in that it enables multiple clients to collaboratively train a global model without sharing raw data in between, thereby facilitating the protection of data privacy. Typically, FL can converge well after a couple of communication rounds, but its convergence is vulnerable to the model poisoning attacks induced by fake clients. Existing works have been devoted to designing various post-processing techniques to alleviate the adverse effects of the model poisoning attacks, they, however, fail to accurately trim off the local models of fake clients while keeping those of benign clients intact during model averaging. In this paper, we investigate the problem of model poisoning attacks in federated learning (FL). To cope with this problem, we design the federated adaptive trimmed mean (FedATM) algorithm, where the clients are sorted in accordance with the distances between local models, and a distance-based threshold is designed to detect the presence of fake clients, thereby preventing the fake local models from destroying the accuracy of model averaging. Simulation results show that the proposed FedATM algorithm is robust to model poisoning attacks as compared to several comparison schemes under various data heterogeneities.
NTIRE 2023 Challenge on Image Super-Resolution (×4): Methods and Results

This paper reviews the NTIRE 2023 challenge on image super-resolution (×4), focusing on the proposed solutions and results. The task of image super-resolution (SR) is to generate a high-resolution (HR) output from a corresponding low-resolution (LR) input by leveraging prior information from paired LR-HR images. The aim of the challenge is to obtain a network design/solution capable to produce high-quality results with the best performance (e.g., PSNR). We want to explore how high performance we can achieve regardless of computational cost (e.g., model size and FLOPs) and data. The track of the challenge was to measure the restored HR images with the ground truth HR images on DIV2K testing dataset. The ranking of the teams is determined directly by the PSNR value. The challenge has attracted 192 registered participants, where 15 teams made valid submissions. They achieve state-of-the-art performance in single image super-resolution.
DIFT: Dynamic Iterative Field Transforms for Memory Efficient Optical Flow

Recent advancements in neural network-based optical flow estimation often come with prohibitively high computational and memory requirements, presenting challenges in their model adaptation for mobile and low-power use cases. In this paper, we introduce a lightweight low-latency and memory-efficient model, Dynamic Iterative Field Transforms (DIFT), for optical flow estimation feasible for edge applications such as mobile, XR, micro UAVs, robotics and cameras. DIFT follows an iterative refinement framework leveraging variable resolution of cost volumes for correspondence estimation. We propose a memory efficient solution for cost volume processing to reduce peak memory. Also, we present a novel dynamic coarse-to-fine cost volume processing during various stages of refinement to avoid multiple levels of cost volumes. We demonstrate first real-time cost-volume based optical flow DL architecture on Snapdragon 8 Gen 1 HTP efficient mobile AI accelerator with 32 inf/sec and 5.89 EPE (endpoint error) on KITTI with manageable accuracy-performance tradeoffs.
Robust Monocular 3D Human Motion with Lasso-Based Differential Kinematics

This work introduces a method to robustly reconstruct 3D human motion from the motion of 2D skeletal landmarks. We propose to use a lasso (least absolute shrinkage and selection operator) optimization framework where the ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> -norm is computed over the vector of differential angular kinematics and the ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> -norm is computed over the differential 2D reprojection error. The ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</inf> -norm term allows us to model sparse kinematic angular motion. The minimization of the reprojection error allows us to assume a bounded noise in both the kinematic model and the 2D landmark detection. This bound is controlled by a scale factor associated to the ℓ <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</inf> -norm data term. A posteriori verification condition is provided to check whether or not the lasso formulation has allowed us to recover the ground-truth 3D human motion. Results on publicly available data demonstrates the effectiveness of the proposed approach on state-of-the-art methods. It shows that both sparsity and bounded noise assumptions encoded in lasso formulation are robust priors to safely recover 3D human motion.
NeRT: Implicit Neural Representations for Unsupervised Atmospheric Turbulence Mitigation

The atmospheric turbulence mitigation problem has emerged as a challenging inverse problem in the communities of computer vision and optics. However, current methods either rely heavily on the quality of the training dataset or fail to generalize over various scenarios, such as static scenes, dynamic scenes, and text reconstructions. We propose a novel implicit neural representation for unsupervised atmospheric turbulence mitigation (NeRT). NeRT leverages the implicit neural representations and the physically correct tilt-then-blur turbulence model to reconstruct the clean and undistorted image, given only dozens of distorted images. Further, we show that NeRT outperforms the state-of-the-art through various qualitative and quantitative evaluations. Lastly, we incorporate NeRT into continuously captured video sequences and demonstrate 48 × speedup.
PanoPoint: Self-Supervised Feature Points Detection and Description for 360° Panorama

We introduce PanoPoint, the joint feature point detection and description applied to the nonlinear distortions and the multi-view geometry problems between 360° panoramas. Our fully convolutional model operates directly in panoramas and computes pixel-level feature point locations and associated descriptors in a single forward pass rather than performing image preprocessing (e.g. panorama to Cubemap) followed by feature detection and description. To train the PanoPoint model, we propose PanoMotion, which simulates the representation between different viewpoints and generates warped panoramas. Moreover, we propose PanoMotion Adaptation, a multi-viewpoint adaptation annotation approach for boosting feature point detection repeatability instead of manual labelling. We train on the annotated synthetic dataset generated by the above method, which outperforms the traditional and other learned approaches and achieves state-of-the-art results on repeatability, localization accuracy, point correspondence precision and real-time metrics, especially for panoramas with significant viewpoint and illumination changes.
SC-NAFSSR: Perceptual-Oriented Stereo Image Super-Resolution Using Stereo Consistency Guided NAFSSR

Stereo image Super-Resolution (SR) has made significant progress since binocular systems are widely accepted in recent years. Most stereo SR methods focus on improving the PSNR performance, while their visual quality is over-smoothing and lack of detail. Perceptual-oriented SR methods are mainly designed for single-view images, thereby their performance decreases on stereo SR due to stereo inconsistency. We propose a perceptual-oriented stereo SR framework that considers both single-view and cross-view information, noted as SC-NAFSSR. With NAF-SSR [3] as our backbone, we combine LPIPS-based perceptual loss and VGG-based perceptual loss for perceptual training. To improve stereo consistency, we perform supervision on each Stereo Cross-Attention Module (SCAM) with stereo consistency loss [27], which calculates photometric loss, smoothness loss, and cycle loss using the cycle-attention maps and valid masks of SCAM. Furthermore, we propose training strategies to fully exploit the performance on perceptual-oriented stereo SR. Both extensive experiments and ablation studies demonstrate the effectiveness of our proposed method. In particular, SC-NAFSSR outperforms the SOTA methods on Flickr1024 dataset [30]. In the NTIRE 2023 Stereo Image Super-Resolution Challenge Track 2 Perceptual & Bicubic [26], SC-NAFSSR ranked 2nd place on the leaderboard. Our source code is available at https://github.com/FVL2020/SC-NAFSSR.
Privacy-Preserving Data Aggregation in IoTs: A Randomize-then-Shuffle Paradigm

Aiming at providing a more rigorous data safeguard for the Internet of Things (IoTs), this paper initiates the study of privacy-preserving data aggregation. We propose a "randomize-then-shuffle" paradigm, which can be generalized into a two-step procedure, that is, a noise addition step plus a random permutation step. More specially, we design an efficient randomizer, which carefully guides the Data Contributors (DCs) to choose the privacy level and obfuscates the truth to ensure local differential privacy. Then, a shuffler is employed to receive the noisy data from all DCs. After that, it breaks the correct linkage between the senders and the receivers by applying a random permutation. Extensive simulations are provided to explore the privacy-utility landscape of our proposed paradigm.
Online Distillation with Continual Learning for Cyclic Domain Shifts

In recent years, online distillation has emerged as a powerful technique for adapting real-time deep neural networks on the fly using a slow, but accurate teacher model. However, a major challenge in online distillation is catastrophic forgetting when the domain shifts, which occurs when the student model is updated with data from the new domain and forgets previously learned knowledge. In this paper, we propose a solution to this issue by leveraging the power of continual learning methods to reduce the impact of domain shifts. Specifically, we integrate several state-of-the-art continual learning methods in the context of online distillation and demonstrate their effectiveness in reducing catastrophic forgetting. Furthermore, we provide a detailed analysis of our proposed solution in the case of cyclic domain shifts. Our experimental results demonstrate the efficacy of our approach in improving the robustness and accuracy of online distillation, with potential applications in domains such as video surveillance or autonomous driving. Overall, our work represents an important step forward in the field of online distillation and continual learning, with the potential to significantly impact real-world applications.
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes

Training models to apply linguistic knowledge and visual concepts from 2D images to 3D world understanding is a promising direction that researchers have only recently started to explore. In this work, we design a novel 3D pre-training Vision-Language method that helps a model learn semantically meaningful and transferable 3D scene point cloud representations. We inject the representational power of the popular CLIP model into our 3D encoder by aligning the encoded 3D scene features with the corresponding 2D image and text embeddings produced by CLIP. To assess our model’s 3D world reasoning capability, we evaluate it on the downstream task of 3D Visual Question Answering. Experimental quantitative and qualitative results show that our pre-training method outperforms state-of-the-art works in this task and leads to an interpretable representation of 3D scene features.