Reviewerly/exHarmony huggingface dataset:
Best practices of mixed-codec adaptive bitrate streaming

One of the biggest challenges in Meta’s video delivery system is device fragmentation. Due to the large user base of Meta’s Family of Apps, our supported devices range from single-core Galaxy Y to the latest Galaxy S22, from the first generation of iPad mini to iPhone 14 pro. Moreover, for devices that do not support hardware decoding for more advanced codecs, such as AV1, we have to rely on software decoders which require high compute power and memory bandwidth. It would be ideal if we can deliver high resolution AVC or VP9 encoded ABR lanes along with low resolution AV1 encoded ABR lanes for the same video, so the client device can choose which one to play based on its compute capacity. In addition, when user uploaded videos that are already encoded with advanced codecs, such as VP9 or HEVC, while generating ABR encoding ladders from the uploaded video using AVC, we also want to deliver the original uploaded video as passthrough to maximize the quality. In both use cases, we will need to support streaming ABR manifest with multiple encoded bitstreams from different codecs and play them smoothly on the client side. On top of that, we can also optimize the ABR encoding and delivery system to select ABR lanes encoded with different codecs for different bitrate targets. In this paper, we will describe how we implemented the end-to-end mixed codec manifest support and deployed the solution in production. The proposed approach effectively generalizes encoding selection into a filtering pipeline, evaluating device capacity via a feedback loop, and being adaptive to the bandwidth estimation, viewport size and device capacity.
Enhancing the quality of gauge images captured in haze and smoke scenes through deep learning

Images captured in hazy and smoky environments suffer from reduced visibility, posing a challenge when monitoring infrastructures and hindering emergency services during critical situations. The proposed work investigates the use of the deep learning models to enhance the automatic, machine-based readability of gauge in smoky environments, with accurate gauge data interpretation serving as a valuable tool for first responders. The study utilizes two deep learning architectures, FFA-Net and AECR-Net, to improve the visibility of gauge images, corrupted with light up to dense haze and smoke. Since benchmark datasets of analog gauge images are unavailable, a new synthetic dataset, containing over 14,000 images, was generated using the Unreal Engine. The models were trained with an 80% train, 10% validation, and 10% test split for the haze and smoke dataset, respectively. For the synthetic haze dataset, the SSIM and PSNR metrics are about 0.98 and 43 dB, respectively, comparing well to state-of-the art results. Additionally, more robust results are retrieved from the AECR-Net, when compared to the FFA-Net. Although the results from the synthetic smoke dataset are poorer, the trained models achieve interesting results. In general, imaging in the presence of smoke are more difficult to enhance given the inhomogeneity and high density. Secondly, FFA-Net and AECR-Net are implemented to dehaze and not to desmoke images. This work shows that use of deep learning architectures can improve the quality of analog gauge images captured in smoke and haze scenes immensely. Finally, the enhanced output images can be successfully post-processed for automatic autonomous reading of gauges.
Switchable CNNs for in-loop restoration and super-resolution for AV2

In video codecs, CNN-based models have shown huge promise in two related tasks: in-loop restoration and frame super-resolution. In our previous work, we presented a framework that uses a common CNN architecture with downloadable model parameters for both these tasks with a preliminary performance study, where encoderside selection of scale factor was left as future work. The advantage of a common architecture with switchable parameters is that a single hardware inference engine can be utilized in all cases of same-resolution and super-resolution restoration, thereby limiting implementation costs. In this paper, we fully integrate this framework into the under-development AV2 video codec from the Alliance for Open Media (AOM). We also implement an algorithm for encoder-side selection of the super-resolution scale factor. With this implementation, we are able to achieve combined compression improvement up to −3.5% (AI) and −3.9% (RA) in BDRATE PSNR-Y and up to −7.8% (AI) and −7.9% (RA) in BDRATE VMAF, with inference cost as low as 1500 MACs/pixel.
Explanation of face recognition via saliency maps

Despite the significant progress in recent years, deep face recognition is often treated as a “black box” and has been criticized for lacking explainability. It becomes increasingly important to understand the characteristics and decisions of deep face recognition systems to make them more acceptable to the public. Explainable face recognition (XFR) refers to the problem of interpreting why a recognition model matches a probe face with one identity over others. Recent studies have explored use of visual saliency maps as an explanation mechanism, but they often lack a deeper analysis in the context of face recognition. This paper starts by proposing a rigorous definition of explainable face recognition (XFR) which focuses on the decision-making process of the deep recognition model. Based on that definition, a similarity-based RISE algorithm (S-RISE) is then introduced to produce high-quality visual saliency maps for a deep face recognition model. Furthermore, an evaluation approach is proposed to systematically validate the reliability and accuracy of general visual saliency-based XFR methods.
Custom precision accelerators for energy-efficient image-to-image transformations in motion picture workflows

Image to Image (I2I) transformations have been an integral part of video processing workflows with applications in Image Synthesis for Virtual Productions, Segmentation, and Matting, among others. Over the years, deep learning-based approaches have been enabling new methods and tools for automating parts of the processing pipeline, reducing the human effort involved in post-production workflows. These compute-intensive models are often accelerated through on-premise or in-cloud GPU instances to improve the responsiveness and latency while expending large amounts of energy in performing these complex transformations. In this work, we present an approach for optimising the energy efficiency of I2I deep-learning models using quantised neural networks accelerated on a server-style FPGA. We use deep learning-based alpha background matting as the I2I application which is implemented using a U-Net conditional Generative Adversarial Network deep learning model. The model is trained and quantised using Vitis-AI flow from AMD/Xilinx and deployed on a data centre class Alveo U50 FPGA device. Our results show that the quantised model on the FPGA achieves a 1.14× higher throughput for inference acceleration while consuming 11× lower energy consumption per inference when compared to a GPU-accelerated version of the model on a 3080-Ti, while generating nearly identical results with an average IoU &gt; 0.95 across multiple user images at 1080p and 4K resolutions. Additionally, offloads to the FPGA device can be seamlessly integrated into widely used motion picture tools like NUKE with minimal effort. With most cloud providers integrating heterogenous platforms (including FPGAs) into systems, we envision that this work paves the way for more efficient utilisation of custom precision deep-learning models and FPGA acceleration in deep learning-based motion picture workflows.
Efficient video processing at scale using MSVP

Videos uploaded to Meta's Family-of-Apps are transcoded into multiple bitstreams of various codec formats, resolutions and quality to provide the best video quality across the wide variety of devices and connection bandwidth constraints. On Facebook alone, there are more than 4 billion video views per day and to address the video processing at this scale, we needed a video processing solution that can deliver the best video quality possible, with the shortest amount of encoding time — all while being energy efficient, programmable, and scalable. In this paper, we present, Meta Scalable Video Processor (MSVP) that can do video processing at on-par quality compared to SW solutions but at a small fraction of the compute time and energy. Each MSVP ASIC can offer a peak SIMO (Single Input Multiple Output) transcoding performance of 4K at 15fps at the highest quality configuration and can scale up to 4K at 60fps at the standard quality configuration. This performance is achieved at ~10W of PCIe module power. We achieved a throughput gain of ~9x for H.264 when compared against libx264 SW encoding. For VP9, we achieved a throughput gain of ~50x when compared with libVPX speed 2 preset. Key components of MSVP transcoding include video decode, scalar, encoding and quality metric computation. In this paper, we go over ASIC architecture of MSVP, design of individual components and compare the perf/W vs quality against standard industry used SW encoders.
Next-generation manufacturing: achieving scalable edge inference for image analytics solutions

Image analytics solutions are essential for next generation manufacturing as they can compute important KPIs related to production, maintenance, and quality in real-time which are otherwise not obtainable through machine data. One main challenge that reduces the performance of an image analytics model is when, during inference, it encounters an image, whose statistical characteristics are very different from the images that it had been trained on. This is called data drift. In commercial applications this is addressed by forming a very large dataset such as ImageNet, which contains several images corresponding to the same class and then training a model over it so hence there is little to no drift during inference. However, image analytics solutions in factories are usually custom developed for a specific situation such as one line/product which does scale when the situation changes due to drift which can be caused due to changes in lighting, camera placement, different camera specs in a new line, dust or oil on camera lens, occlusion by human workers and a multitude of other reasons. For factory settings it is not possible to create an ImageNet type dataset given the sheer volume of different and moving parts within a factory shopfloor, some of which are very specific to a given factory. Instead monitoring and compensating this data drift will detect and resolve the degradation in image analytics model performance. In this paper, the proposed solution detects the drift in images during inference using convolutional variational autoencoder and compensates this drift with minimum system integration to easily scale the solution for wide range of changing conditions.
Phase retrieval by pattern classification and circular mean for robust optical testing

For accurate optical testing of large mirrors with long radii of curvature, multiple measurements using single-shot phaseshifting interferometers are typically required to reduce the impact of air turbulence on air stratification prevention. This paper proposes a new technique that utilizes hierarchical clustering to classify acquired wrapped phases into similar pattern clusters, eliminating the influence of unknown noises by excluding minority patterns. For each cluster, the circular mean is used to calculate denoised wrapped phases, and the surface figure is obtained from the unwrapped phases. Since the number of phase-unwrapping algorithm runs equals the number of chosen clusters, the proposed technique is significantly faster than conventional procedures. In 500 measurements on a 1.5 m diameter spherical mirror, the proposed technique reduced measurement time and the root-mean-square error's standard deviation of the surface figure by approximately 23% and 15%, respectively, compared to conventional procedures.
Reference Management Software

Bibliographic reference managers are specialized software tools whose objective is to support the management of bibliographic data by the end user. The functionalities they offer are designed to facilitate the capture, organization, management, and exploitation of bibliographic references. Their main purpose is to help the researcher in the scientific and technical publication processes, but the latest developments are promoting the publication and collaborative use of reference data.
e-LIS ePrints in Library and Information Science

e-LIS (E-prints in Library and Information Science) is an international open access archive for research articles and papers in the field of library and information science (LIS). It was established in 2003 and is managed by a team of volunteers, with the aim of providing free access to quality research in LIS to researchers, scholars, and practitioners all over the world. e-LIS contains a wide range of research papers, conference proceedings, book chapters, and other materials in the field of LIS, covering topics such as information management, knowledge organization, digital libraries, archives and preservation. e-LIS hosts over 24,000 documents, in 27 languages. All the continents are represented, involving over 60 different countries. Users can browse the archive and search for specific documents using keywords, authors, titles, or subject headings. e-LIS provides usage statistics and other helpful features to support research and scholarship. Overall, e-LIS plays an important role in advancing research and promoting the exchange of knowledge and ideas in the LIS field.
Library Technology Hardware

Libraries around the world use various computer and other information technology to fulfill their respective missions. Throughout the past fifty years, the technologies and related hardware have shifted with ever increasing technology landscape. These changes have been greatly impacted by the rise of the internet and the World Wide Web. Examining the historical context and evolution of library technology hardware is important in understanding how and why technologies and their related hardware are used in libraries today. The library of today may include a wide range of technologies including desktops, laptops, servers, printers, scanners, and other use case specific technologies.
Library Software, an Overview

Libraries are institutions that organize information. As such, libraries have created and used various technologies including software throughout their history to optimize organization. Because libraries are information institutions with core values that include recognizing the need to save time for users and the need to always evolve, libraries consistently employ technology for operations and for the continuous improvement necessary to operate in a manner that serves users. Libraries have a long relationship with software, and with the standards and systems within which software exists.
JPEG XS profiles and levels for screen content coding

JPEG XS is a lightweight, low-latency image coding standard developed for transmission of video streams over IP. The third edition of JPEG XS, currently under development of ISO, adds a frame buffer and temporal predictive coding in the wavelet domain to improve its coding efficiency on screen content and thus improves JPEG XS for remote desktop applications. Due to bandwidth constraints, the frame buffer itself needs to be compressed, and thus the frame buffer bandwidth needs to be considered for interoperability between implementations. This paper reports on core experiments conducted in JPEG (ISO/IEC SC29 WG1) to design profiles and levels for the third edition, and how WG1 is currently considering profiling the screen content coding extensions of JPEG XS.
Smart Libraries and Library 3.0

This entry describes the smart systems, digital services and Web 3.0 technologies that constitute Smart Libraries and Library 3.0, and explores the roles that virtual spaces and physical places have in this next generation of libraries. Key points include that the Smart Library and Library 3.0 are increasingly user-centric and focused on personalization and content creation as well as collections, and that physical library spaces are becoming much more informal and socially orientated but remain crucially significant. The entry concludes by considering implications of smart services and new digital technologies for library professionals and outlines some projections for the future.
Publishing

Publishing has expanded from the print-only, controlled authorship world to a range of outputs where anyone can be an author and publisher. Publishing outputs can be categorized into two main types of publishing: self-publishing and traditional publishing. In self-publishing, the author initiates and maintains control of the content that is published. In traditional publishing, the publisher absorbs the initial publishing costs and provides services, such as editing, design, production, distribution, and marketing. Whether self- or traditional, a reputable publisher places strong value on the production of high-quality information. Several criteria are recommended to consider when evaluating the reputation of a publisher, including ones related to bias. The large, complex industry of publishing will continue to rapidly evolve with increased globalization and incorporation of new technologies, such as artificial intelligence.
Validation and assessment of a mixed reality solution for enhanced TV viewer engagement

This paper explores the use of Mixed Reality in live television shows by allowing remote participants to “teleport” into a virtual studio. The solution utilizes background extraction (BE) and super-resolution (SR) modules to extract remote participants from their videos and composite them seamlessly into the studio footage, allowing for participation in live TV programs using standard mobile devices or webcams. This paper aims to investigate the impact of capturing devices and background settings on the output videos from the end user’s point of view. The results of the study are presented and discussed with a focus on the effectiveness of the BE and SR components in response to variations in capturing devices and background settings.
Assessing objective video quality in multi-screen video delivery

In today’s online video delivery systems, videos are streamed and displayed on various devices with different screen sizes, from large-screen UHD and HDTVs to smaller-screen devices such as mobile phones and tablets. A video will be perceived differently depending on the device’s screen size, pixel density, and viewing distance when viewed on different devices. Quality models which can estimate the relative differences in perceptual quality of a video on different devices can be used to understand the end-user QoE, design optimal encoding ladders for a multi-screen delivery environment, and better rate-adaptation algorithms. We previously presented a BC-KU Multi-Screen dataset1 consisting of subjective scores for different contents encoded in different resolution-bitrate pairs when viewed on three different devices. This paper presents several contributions extending the earlier dataset, which is of interest to the multimedia quality of experience (QoE) community. We first present an in-depth statistical data analysis on the previously unpublished individual subjective ratings of the Multi-Screen dataset. To better understand the relative differences in MOS scores, we present and analyze various demographic information about the test participants. We then evaluate the performance of twelve quality metrics based on five different performance measures. Individual subjective ratings, analysis scripts, and results are available as an open-source dataset. We believe the newly contributed results, files, and scripts will help analyze and design improved, low-complexity parametric models for multi-screen video delivery systems.
JPEG frequency domain-based image steganography using iterative prediction and decision rules to improved data integration

This paper proposes a novel steganographic method that employs a feedback mechanism to improve the efficiency and stealth of data hiding within the Discrete Cosine Transform (DCT) coefficients of JPEG images. This method enhances the correlation between the hidden message and the cover image, while minimizing the perceptible changes to the image. The system starts by dividing the cover image into blocks and applying DCT to each. It then evaluates the correlation between the hidden message and the DCT coefficients to identify potential data embedding points. A trained decision rules algorithm then chooses the optimal data embedding technique, considering factors like the size and location of the DCT coefficient within image blocks. Different embedding techniques are employed. The system subsequently generates feedback based on metrics such as image quality and data detectability, refining the decision ruls's effectiveness over time. By employing this dynamic approach, our system adaptively improves the data hiding process, enhancing capacity and minimizing detectability. This work opens new doors in the realm of steganography, presenting an intelligent system capable of adaptively embedding data with optimized stealth and efficiency.
Maintaining original color when applying super-resolution to individual bands

We have developed a system that applies deep learning application super-resolution (SR) to multispectral and hyperspectral geospatial satellite imagery to deduce higher resolution images from lower resolution images while maintaining the original color of the lower resolution pixels. A super-resolution model, which uses Deep Convolution Neural Networks (DCNNs), is trained using individual image bands, a large crop size or tile size of 512 &times; 512 pixels, and a de-noise algorithm. Applying our algorithms to maintain the original color of the image bands improves the quality metrics of the super-resolution images as measured by peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) of super-resolution images. One of the most important applications of satellite images is to automatically detect small objects such as vehicles and small boats. With super-resolution images generated by our system, the object detection accuracy (recall and precision) has improved by 20% with Planet® multispectral satellite images.
Improving deepfake detectors against real-world perturbations with amplitude-phase switch augmentation

In recent years, the remarkable progress in facial manipulation techniques has raised social concerns due to their potential malicious usage and has received considerable attention from both industry and academia. While current deep learning-based face forgery detection methods have achieved promising results, their performance often degrades drastically when they are tested in non-trivial situations under realistic perturbations. This paper proposes to leverage the information in the frequency domain, particularly the phase spectrum, to better differentiate between deepfakes and authentic images. Specifically, a new augmentation method called degradation-based amplitude-phase switch (DAPS) is proposed, which disregards the sensitive amplitude spectrum of a forged facial image and enforces the detection network to focus on phase components during the training process. Extensive evaluation results from a realistic assessment framework show that the proposed augmentation method significantly improves the robustness of two deepfake detectors analyzed and consistently outperform other augmentation approaches under various perturbations.