Reviewerly/exHarmony huggingface dataset:
NeRFVS: Neural Radiance Fields for Free View Synthesis via Geometry Scaffolds

We present NeRFVS, a novel neural radiance fields (NeRF) based method to enable free navigation in a room. NeRF achieves impressive performance in rendering images for novel views similar to the input views while suffering for novel views that are significantly different from the training views. To address this issue, we utilize the holistic priors, including pseudo depth maps and view coverage information, from neural reconstruction to guide the learning of implicit neural representations of 3D indoor scenes. Concretely, an off-the-shelf neural reconstruction method is leveraged to generate a geometry scaffold. Then, two loss functions based on the holistic priors are proposed to improve the learning of NeRF: 1) A robust depth loss that can tolerate the error of the pseudo depth map to guide the geometry learning of NeRF; 2) A variance loss to regularize the variance of implicit neural representations to reduce the geometry and color ambiguity in the learning procedure. These two loss functions are modulated during NeRF optimization according to the view coverage information to reduce the negative influence brought by the view coverage imbalance. Extensive results demonstrate that our NeRFVS outperforms state-of-the-art view synthesis methods quantitatively and qualitatively on indoor scenes, achieving high-fidelity free navigation results.
A Characteristic Function-Based Method for Bottom-Up Human Pose Estimation

Most recent methods formulate the task of human pose estimation as a heatmap estimation problem, and use the overall L2 loss computed from the entire heatmap to optimize the heatmap prediction. In this paper, we show that in bottom-up human pose estimation where each heatmap often contains multiple body joints, using the overall L2 loss to optimize the heatmap prediction may not be the optimal choice. This is because, minimizing the overall L2 loss cannot always lead the model to locate all the body joints across different sub-regions of the heatmap more accurately. To cope with this problem, from a novel perspective, we propose a new bottom-up human pose estimation method that optimizes the heatmap prediction via minimizing the distance between two characteristic functions respectively constructed from the predicted heatmap and the groundtruth heatmap. Our analysis presented in this paper indicates that the distance between these two characteristic functions is essentially the upper bound of the L2 losses w.r.t. sub-regions of the predicted heatmap. Therefore, via minimizing the distance between the two characteristic functions, we can optimize the model to provide a more accurate localization result for the body joints in different sub-regions of the predicted heatmap. We show the effectiveness of our proposed method through extensive experiments on the COCO dataset and the CrowdPose dataset.
Learning Semantic-Aware Disentangled Representation for Flexible 3D Human Body Editing

3D human body representation learning has received increasing attention in recent years. However, existing works cannot flexibly, controllably and accurately represent human bodies, limited by coarse semantics and unsatisfactory representation capability, particularly in the absence of supervised data. In this paper, we propose a human body representation with fine-grained semantics and high reconstruction-accuracy in an unsupervised setting. Specifically, we establish a correspondence between latent vectors and geometric measures of body parts by designing a part-aware skeleton-separated decoupling strategy, which facilitates controllable editing of human bodies by modifying the corresponding latent codes. With the help of a bone-guided auto-encoder and an orientation-adaptive weighting strategy, our representation can be trained in an unsupervised manner. With the geometrically meaningful latent space, it can be applied to a wide range of applications, from human body editing to latent code interpolation and shape style transfer. Experimental results on public datasets demonstrate the accurate reconstruction and flexible editing abilities of the proposed method. The code will be available at http://cic.tju.edu.cn/faculty/likun/projects/SemanticHuman.
Scene-Aware Egocentric 3D Human Pose Estimation

Egocentric 3D human pose estimation with a single head-mounted fisheye camera has recently attracted attention due to its numerous applications in virtual and augmented reality. Existing methods still struggle in challenging poses where the human body is highly occluded or is closely interacting with the scene. To address this issue, we propose a scene-aware egocentric pose estimation method that guides the prediction of the egocentric pose with scene constraints. To this end, we propose an egocentric depth estimation network to predict the scene depth map from a wide-view egocentric fisheye camera while mitigating the occlusion of the human body with a depth-inpainting network. Next, we propose a scene-aware pose estimation network that projects the 2D image features and estimated depth map of the scene into a voxel space and regresses the 3D pose with a V2V network. The voxel-based feature representation provides the direct geometric connection between 2D image features and scene geometry, and further facilitates the V2V network to constrain the predicted pose based on the estimated scene geometry. To enable the training of the aforementioned networks, we also generated a synthetic dataset, called EgoGTA, and an in-the-wild dataset based on EgoPW, called EgoPW-Scene. The experimental results of our new evaluation sequences show that the predicted 3D egocentric poses are accurate and physically plausible in terms of human-scene interaction, demonstrating that our method outperforms the state-of-the-art methods both quantitatively and qualitatively.
Token Boosting for Robust Self-Supervised Visual Transformer Pre-training

Learning with large-scale unlabeled data has become a powerful tool for pre-training Visual Transformers (VTs). However, prior works tend to overlook that, in real-world scenarios, the input data may be corrupted and unreliable. Pre-training VTs on such corrupted data can be challenging, especially when we pre-train via the masked autoencoding approach, where both the inputs and masked “ground truth” targets can potentially be unreliable in this case. To address this limitation, we introduce the Token Boosting Module (TBM) as a plug-and-play component for VTs that effectively allows the VT to learn to extract clean and robust features during masked autoencoding pre-training. We provide theoretical analysis to show how TBM improves model pre-training with more robust and generalizable representations, thus benefiting down stream tasks. We conduct extensive experiments to analyze TBM's effectiveness, and results on four corrupted datasets demonstrate that TBM consistently improves performance on downstream tasks.
DeCo: Decomposition and Reconstruction for Compositional Temporal Grounding via Coarse-to-Fine Contrastive Ranking

Understanding dense action in videos is a fundamental challenge towards the generalization of vision models. Several works show that compositionality is key to achieving generalization by combining known primitive elements, especially for handling novel composited structures. Compositional temporal grounding is the task of localizing dense action by using known words combined in novel ways in the form of novel query sentences for the actual grounding. In recent works, composition is assumed to be learned from pairs of whole videos and language embeddings through large scale self-supervised pre-training. Alternatively, one can process the video and language into word-level primitive elements, and then only learn fine-grained semantic correspondences. Both approaches do not consider the granularity of the compositions, where different query granularity corresponds to different video segments. Therefore, a good compositional representation should be sensitive to different video and query granularity. We propose a method to learn a coarse-to-fine compositional representation by decomposing the original query sentence into different granular levels, and then learning the correct correspondences between the video and recombined queries through a contrastive ranking constraint. Additionally, we run temporal boundary prediction in a coarse-to-fine manner for precise grounding boundary detection. Experiments are performed on two datasets, Charades-CG and ActivityNet-CG, showing the superior compositional generalizability of our approach.
SeSDF: Self-Evolved Signed Distance Field for Implicit 3D Clothed Human Reconstruction

We address the problem of clothed human reconstruction from a single image or uncalibrated multiview images. existing methods struggle with reconstructing detailed geometry of a clothed human and often require a calibrated setting for multiview reconstruction. We propose a flexible framework which, by leveraging the parametric SMPL-X model, can take an arbitrary number of input images to reconstruct a clothed human model under an uncalibrated setting. At the core of our framework is our novel self-evolved signed distance field (SeSDF) module which allows the framework to learn to deform the signed distance field (SDF) derived from the fitted SMPL-X model, such that detailed geometry reflecting the actual clothed human can be encoded for better reconstruction. Besides, we propose a simple method for self-calibration of multiview images via the fitted SMPL-X parameters. This lifts the requirement of tedious manual calibration and largely increases the flexibility of our method. Further, we introduce an effective occlusion-aware feature fusion strategy to account for the most useful features to reconstruct the human model. We thoroughly evaluate our framework on public benchmarks, demonstrating significant superiority over the state-of-the-arts both qualitatively and quantitatively.
B-Spline Texture Coefficients Estimator for Screen Content Image Super-Resolution

Screen content images (SCIs) include many informative components, e.g., texts and graphics. Such content creates sharp edges or homogeneous areas, making a pixel distribution of SCI different from the natural image. Therefore, we need to properly handle the edges and textures to minimize information distortion of the contents when a display device's resolution differs from SCIs. To achieve this goal, we propose an implicit neural representation using B-splines for screen content image super-resolution (SCI SR) with arbitrary scales. Our method extracts scaling, translating, and smoothing parameters of B-splines. The followed multilayer perceptron (MLP) uses the estimated B-splines to recover high-resolution SCI. Our network outperforms both a transformer-based reconstruction and an implicit Fourier representation method in almost upscaling factor, thanks to the positive constraint and compact support of the B-spline basis. Moreover, our SR results are recognized as correct text letters with the highest confidence by a pre-trained scene text recognition network. Source code is available at https://github.com/ByeongHyunPak/btc.
Semi-Supervised Video Inpainting with Cycle Consistency Constraints

Deep learning-based video inpainting has yielded promising results and gained increasing attention from re-searchers. Generally, these methods assume that the cor-rupted region masks of each frame are known and easily ob-tained. However, the annotation of these masks are labor-intensive and expensive, which limits the practical application of current methods. Therefore, we expect to relax this assumption by defining a new semi-supervised inpainting setting, making the networks have the ability of completing the corrupted regions of the whole video using the anno-tated mask of only one frame. Specifically, in this work, we propose an end-to-end trainable framework consisting of completion network and mask prediction network, which are designed to generate corrupted contents of the current frame using the known mask and decide the regions to be filled of the next frame, respectively. Besides, we introduce a cycle consistency loss to regularize the training parameters of these two networks. In this way, the completion network and the mask prediction network can constrain each other, and hence the overall performance of the trained model can be maximized. Furthermore, due to the natural existence of prior knowledge (e.g., corrupted contents and clear bor-ders), current video inpainting datasets are not suitable in the context of semi-supervised video inpainting. Thus, we create a new dataset by simulating the corrupted video of real-world scenarios. Extensive experimental results are reported to demonstrate the superiority of our model in the video inpainting task. Remarkably, although our model is trained in a semi-supervised manner, it can achieve compa-rable performance as fully-supervised methods.
AnyFlow: Arbitrary Scale Optical Flow with Implicit Neural Representation

To apply optical flow in practice, it is often necessary to resize the input to smaller dimensions in order to reduce computational costs. However, downsizing inputs makes the estimation more challenging because objects and motion ranges become smaller. Even though recent approaches have demonstrated high-quality flow estimation, they tend to fail to accurately model small objects and precise boundaries when the input resolution is lowered, restricting their applicability to high-resolution inputs. In this paper, we introduce AnyFlow, a robust network that estimates accurate flow from images of various resolutions. By representing optical flow as a continuous coordinate-based representation, AnyFlow generates outputs at arbitrary scales from low-resolution inputs, demonstrating superior performance over prior works in capturing tiny objects with detail preservation on a wide range of scenes. We establish a new state-of-the-art performance of cross-dataset generalization on the KITTI dataset, while achieving comparable accuracy on the online benchmarks to other SOTA methods.
Bitstream-Corrupted JPEG Images are Restorable: Two-stage Compensation and Alignment Framework for Image Restoration

In this paper, we study a real-world JPEG image restoration problem with bit errors on the encrypted bitstream. The bit errors bring unpredictable color casts and block shifts on decoded image contents, which cannot be resolved by existing image restoration methods mainly relying on pre-defined degradation models in the pixel domain. To address these challenges, we propose a robust JPEG decoder, followed by a two-stage compensation and alignment framework to restore bitstream-corrupted JPEC images. Specifically, the robust JPEC decoder adopts an error-resilient mechanism to decode the corrupted JPEG bitstream. The two-stage framework is composed of the self-compensation and alignment (SCA) stage and the guided-compensation and alignment (GCA) stage. The SCA adaptively performs block-wise image color compensation and alignment based on the estimated color and block offsets via image content similarity. The GCA leverages the extracted low-resolution thumbnail from the JPEG header to guide full-resolution pixel-wise image restoration in a coarse-to-fine manner. It is achieved by a coarse-guided pix2pix network and a refine-guided bi-directional Laplacian pyramid fusion network. We conduct experiments on three benchmarks with varying degrees of bit error rates. Experimental results and ablation studies demonstrate the superiority of our proposed method. The code will be released at https://github.com/wenyang001/Two-ACIR.
Research on Digital Image Information Security and Encryption Technology

The development mode of digital and Internet communication has contributed to the phenomenon of taking digital image as the information transmission carrier. Therefore, the transmission network and sending and receiving equipment of digital images are extremely easy to receive external intrusion and attacks, causing information leakage and other information security accidents. On the basis of the digital image features, the digital graphic encryption method and the defects of single feature encryption, further study the chaotic encryption technique of multiple angle encryption. Constructing a modified tesseract transform using chaotic sequence and mirror transform to dislocate the image, and performing a cyclic dissimilarity between the gray values of each pixel point and chaotic sequence to achieve diffusion and confusion between pixel points increases the complexity of the key and increases the number of keys. It makes the time required to crack increase. The algorithm design is simple and easy to implement, and the experimental simulation results conclude that the algorithm has the advantages of easy implementation, sensitivity to the perturbation of the initial conditions of the system, and large key space compared with the traditional chaotic encryption algorithm.
The Effect of Channel Size on Performance of 1D CNN Architecture for Automatic Detection of Self-Reported COVID-19 Symptoms on Twitter

Social media has a crucial role as the most generally accessed source of information by the public for obtaining information on various topics, such as COVID-19 and natural disasters. Using Twitter messages can be beneficial in tasks like detecting symptoms of COVID-19. The results of COVID-19 symptom detection can be used as a benchmark by relevant parties to create policies that can be useful in the future. However, social media data contains various topics, requiring various classification approaches to obtain the most accurate results. The classification was done using variations of word embedding (Word2vec, fastText, and a combination of Word2vec and fastText) and variations of CNN architecture (single-channel and multi-channel). The best accuracy result was obtained by combining fastText and fourth channels, which was 88.5%. The comparison results showed both the single-channel and fourth-channel architectures achieved the highest average accuracy of 86.33%.
Image Watermarking Algorithm Based on q-logarithm Component

A new embedding component for image watermarking in the spatial domain is presented in this paper as a solution to copyright protection and to verify the real owner of digital image. The embedding component called q-logarithm is extracted from a host color image, and is used directly to embed watermark information. In the extraction process of the embedded watermark, the original version of non-watermarked component is first estimated from the average value of watermarked components in a small image area, so that the blind watermark extraction can be achieved. The watermarked image’s quality is assessed by weight Peak Signal to Noise Ratio (wPSNR), and the extracted watermark’s accuracy is assessed by Bit Correction Error (BCR). Investigations for optimal parameters are carried out to obtain q value that gives the most accurate extracted watermark. In the experiments, the performance of the proposed method is compared to the relevant three previous methods. A well-known benchmark called Stirmark is also used to assess the robustness of the embedded watermark. It is shown accordingly that the proposed method can achieve better performance in terms of accuracy and robustness even under attacks.
Annotated Flickr dataset for identification of professional photographers

We collected and computed various data and statistics from a sample of Flickr users who uploaded photos to the platform in December 2021 and their photos, obtaining a final number of 27,516 users and 2,647,928 photos. Having the total number of photos uploaded and the number of photos uploaded in December by each user, we selected a representative sample of those whose activity was not overly concentrated in December and obtained data from those who specified their occupation. In addition to the data collected directly from Flickr, we enriched the dataset with new features resulting from the automated analysis of the photos and their comments. One of the most valuable features of this data collection is that each photo has three Image Quality Assessment scores representing aesthetic and technical aspects. For this, we used Convolutional Neural Networks trained with human-labeled data. Furthermore, we added labels to indicate whether the user is a professional photographer, so the data are specially prepared for supervised training.
Towards democratization of digital twins: Design principles for transformation into a human-building interface

While legacy building digital twins (DTs), such as building management systems, help building and facility managers to make informed decisions, the real end-users of buildings, i.e., occupants, have been considered as data informants without agency for access to and interaction with DT systems. This is due to the lack of multi-user interfaces for occupants to interact with building systems effectively with sufficient observability and controllability. The research on the design of multi-user interfaces has been limited to specialized services in public, shared spaces, or office-specific experiments and fails to produce detailed design principles. In this study, we explore the “democratization of DTs” as an occupant-building interface that invites occupants to monitor the building- and human-related data and make informed decisions. An occupant participatory approach was employed, consisting of a survey, an experience sampling method, and a co-design workshop, to understand occupants' current experiences with environmental controls and desired methods to interact with a DT system. The findings identified five interaction dimensions between occupants and a novel occupant-in-the-loop DT system that provides occupants with observability and controllability over building systems. Accordingly, we establish design principles for DT systems as an occupant-building interface that supports occupants’ dynamic contexts, intuitive controls, collaborative and balanced decision-making between personal comfort and energy consumption, as well as authority and data privacy.
Parameter Estimation of Polynomial Phase Signals Using Spectrum Distinction

The spectrum phase unwrapping algorithm is a recently proposed algorithm for estimating parameters of polynomial phase signals (PPS). It can handle special PPSs known as strictly monotonic PPSs (SMPPS) with a low computational load. However, due to the use of phase differentiation, this method cannot be used for non-SMPPS when signal-to-noise-ratio (SNR) is low. We analyze the spectrum of SMPPS and non-SMPPS. By considering the spectrum distinctions between them, we propose to extend the spectrum phase unwrapping algorithm by replacing the phase differentiation procedure with a one-dimensional search algorithm to handle general PPS and non-polynomial phase signals. Simulation shows that this modification can improve the algorithm's SNR threshold for general PPS and non-polynomial phase signals without considerably increasing the computational complexity.