dataesr/scientific-paragraphs-categorization huggingface dataset:
Furthermore, directly supervising our method using a multi-view and multi-lit light-stage dataset allows us to produce significantly more photorealistic results than PIE  [Tewari et al. 2020b ]. Our method can additionally edit at a faster speed, using just a single feedforward pass, and also edit both illumination and pose simultaneously, unlike PIE. Compared to traditional relighting approaches  [Sun et al. 2020; Zhou et al. 2019] , we obtain higher-quality results as well as allow for changing the camera pose. In summary we make the following contributions: • We combine the strength of supervised learning and generative adversarial modeling in a new way to develop a technique for high-quality editing of scene illumination and camera pose in portrait images. Both properties can be edited simultaneously.
Their method, however, is limited to reproducing the specific face that was captured. Weyrich et al. [2006]  extend this concept using a setup with a much larger number of cameras (  16 ) and a reconstruction pipeline that extracts geometry and a detailed spatially-varying BRDF (SVBRDF) of a face. By scanning hundreds of subjects that way, they extract generalisable statistical information on appearance traits depending on age, gender and ethnicity. The generative power of the extracted quantities, however, is heavily constrained, and examples of sematic appearance editing were limited to subjects from within their face database. In our work, we revisit their original dataset using a state-of-the-art learning framework. Another category of methods tries to infer geometry and reflectance properties from single, unconstrained images.
2019; Nestmeyer et al. 2020; Sun et al. 2019; Wang et al. 2020; Zhang et al. 2020; Zhou et al. 2019]. Nestmeyer et al. [2020]  used a lightstage dataset to train a model that explicitly regresses a diffuse reflectance, as well as a residual component which accounts for specularity and other effects. Similarly,  Wang et al. [2020]  used a light-stage dataset to compute the ground truth diffuse albedo, normal, specularity and shadow images. A network is trained to regress each of these components which are then used in another network to finally relight the portrait image. Instead of explicitly estimating the different reflectance components, methods such as  Sun et al. [2019] ;  Zhou et al. [2019]  directly regress the relighted version of the portrait given the imput image and target illumination.
Here, the target illumination is parameterised either in the form of environment map  [Sun et al. 2019]  or spherical harmonics  [Zhou et al. 2019]. While  Sun et al. [2019]  used light-stage data to obtain their ground truth for supervised learning,  Zhou et al. [2019]  used a ratio image-based approach to generate synthetic training data. Recently,  Zhang et al. [2020]  proposed a method to remove harsh shadows from a monocular portrait image. They created a synthetic data from in-the-wild images by augmenting shadows and training a network to remove these shadows. Using a light-stage dataset, another network is trained to smooth the artifacts that could remain pSpNet pSpNet StyleGAN PhotoAppNet Fig. 2. Our method allows for editing the scene illumination   and camera pose   in an input source image.
We learn to map the StyleGAN  [Karras et al. 2020 ] latent code   of the source image, estimated using pSpNet  [Richardson et al. 2020]  to the latent code   of the output image. StyleGAN  [Karras et al. 2020 ] is then used to synthesis the final output. Our method is trained in a supervised manner using a light-stage dataset with multiple cameras and light sources. For training, we use a latent loss and a perceptual loss defined using a pretrained network. Supervised learning in the latent space of StyleGAN allows for high-quality editing which can generalise to in-the-wild images. Portrait images are from  Weyrich et al. [2006]  and environment map is from  [Gardner et al. 2017; Hold-Geoffroy et al. 2019]. from the first network. While the methods of  [Meka et al. 2019; Nestmeyer et al. 2020; Sun et al.
2019; Wang et al. 2020; Zhang et al. 2020; Zhou et al. 2019 ] can produce high-quality relighting results, they either focus on shadow removal  [Zhang et al. 2020] , or limited by spherical-harmonics illumination representation  [Zhou et al. 2019 ]. In addition, methods trained on light-stage or synthetic datasets struggle to generalise to in-the-wild. They are also limited to only relighting, as they cannot change the camera viewpoint. There are several methods for editing the head pose of portrait images  [Averbuch-Elor et al. 2017; Geng et al. 2018; Kim et al. 2018; Nagano et al. 2018; Siarohin et al. 2019; Wiles et al. 2018 ]. While  Kim et al. [2018]   Recently,  Tewari et al. [2020b]  proposed PIE, a method which can relight, change expressions and synthesise novel views of the portrait image using a generative model.
METHOD Our method takes as input an in-the-wild portrait image, a target illumination and the target camera pose. The output is a portrait image of the same identity, synthesised with the target camera and lit by the target illumination. Given a light-stage dataset of multiple independent illumination sources and viewpoints, the naive approach could be to learn the transformations directly in image space. Instead, we propose to learn the mapping in the latent space of StyleGAN  [Karras et al. 2020 ]. We show that learning using this latent representation helps in generalisation to in-the-wild images with high photorealism. We use StyleGAN2 in our implementation, referred to as StyleGAN for better comprehension.
Dataset We make use of a light-stage  [Weyrich et al. 2006 ] dataset for training our solution. This dataset contains 341 identities captured with 8 different cameras placed in the frontal hemisphere of the face. The camera poses available are shown in Fig. 3. The dataset also contains 150 light source evenly distributed on the sphere. Using this setup, each image is captured with one-light-at-a-time (OLAT) light. Given 150 OLAT images of a person with a specific camera pose, we can linearly combine them using an environment map to obtain relight portrait images  [Debevec et al. 2000 ]. We use 205 HDR environment maps from the Laval Outdoor  [Hold-Geoffroy et al. 2019 ] and 2233 from the Naval Indoor  [Gardner et al. 2017 ] dataset for generating naturally lit images. Camera poses for the images are estimated using the approach of  Yang et al.
[2020]  (pSpNet in Fig. 2 ). The latent space used is 18×512 dimensional, corresponding to the W+ space of StyleGAN. The output of PhotoAppNet is a displacement to the input in the StyleGAN latent space. This is then added to the input latent code to compute   , which is used by StyleGAN to generate the output image. We only train PhotoAppNet, while pSpNet and StyleGAN are pretrained and fixed. We use an MLP-based architecture with a single hidden layer of length 512. ReLU activation is used after the hidden layer. We use independent networks for each of the 18 latent vectors of length 512 corresponding to different resolutions. This is motivated by the design of the StyleGAN network where each 512 dimensional latent code controls a different frequency of image features. The output of each independent network is the output latent code corresponding to the same resolution.
Loss Function We use multiple loss terms to train our network. L (𝐼 𝑡 , 𝐿 𝑡 , Î 𝑡 , L𝑡 , 𝜃 n ) =L l (𝐿 𝑡 , L𝑡 , 𝜃 n ) + L p (𝐼 𝑡 , Î 𝑡 , 𝜃 n ). (1) Here,   denotes the network parameters of PhotoAppNet. Both terms are weighed equally. The first term is a StyleGAN latent loss defined as L l (𝐿 𝑡 , L𝑡 , 𝜃 n ) = ∥𝐿 𝑡 -L𝑡 ∥ 2 2. It enforces the StyleGAN latent code of the output image   to be close to the ground truth latent code L. The second term is a perceptual loss defined as L p (𝐼 𝑡 , Î 𝑡 , 𝜃 n ) = ∥𝜙 (𝐼 𝑡 ) -𝜙 ( Î 𝑡 )∥ 2 2. Here, we employ the learned perceptual similarity metric LPIPS  [Zhang et al. 2018 ]. An ℓ 2 loss is used to compare the AlexNet  [Krizhevsky et al. 2012 ] features  () of the synthesised output and the ground truth images.
Network Training We implement our method in PyTorch and optimise for the weights of PhotoAppNet by minimising the loss function in Eq. 1. We use Adam solver with a learning rate of 0. 0001 and default hyperparameters. As mentioned earlier, the StyleGAN encoder (pSpNet) and generator  [Karras et al. 2020; Richardson et al. 2020 ] are pretrained and fixed during training. We optimise over our training set samples using a batch size equal to 1. Since in-the-wild images are very different from the light-stage data, it is difficult to assess convergence using a light-stage validation dataset. As such, we train our networks using an in-the-wild validation set using qualitative evaluations. Our network take around 10 hours to train on a single NVIDIA Quadro RTX 8000 GPU.
Discussion Existing image-based relighting approaches such as  Sun et al. [2020] ;  Zhou et al. [2019]  rely on much larger trainable networks with several loss functions, such as losses on the input environment map and adversarial losses. Approaches for pose editing such as  Kim et al. [2018] ;  Siarohin et al. [2019] ;  Thies et al. [2019]  rely on conditional generative networks trained with a combination of photometric and adversarial losses. Since we rely on a pretrained generator as our backend renderer, our training is much simpler than existing approaches. We do not need an adversarial loss as the pretrained generator already synthesises results at a high quality. As such, our training is more stable than approaches operating in image space. In addition. the StyleGAN latent representation allows for generalisation with high-quality, even when trained on a dataset with as little as 3 identities (Sec.
[2019]. The images are resized to a resolution of 1024x1024. Since we need the ground truth images for quantitative evaluations, we use the test portion of our lightstage dataset composed of images of 41 identities unseen during training. We create two test sets, Set1 has the input and ground truth pairs captured from the same viewpoint while Set2 includes pairs captured from different viewpoints. The HDR environment maps, randomly sampled from the Naval Outdoor and Naval Indoor datasets  [Gardner et al. 2017; Hold-Geoffroy et al. 2019 ] are used to synthesise the pairs with natural illumination conditions. Viewpoints are randomly sampled from the 8 cameras of the light-stage setup. The input and ground truth images are computed using the same environment map in Set2 for evaluating the viewpoint editing results, while the pairs in Set1 use different environment maps for relighting evaluations.
High-Fidelty Appearance Editing Figs. 4, 5,  and 6  show simultaneous viewpoint and illumination editing results of our method for various subjects. We also show the StyleGAN projection of the input images estimated by  Richardson et al. [2020]. Our approach produces high-quality photorealistic results and synthesises the full portrait, including hair, eyes, mouth, torso and the background, while preserving the identity, expression and other properties (such as facial hair). Our method works well on people of different races. Additionally, the results show that our method can preserve a variety of reflectance properties, resulting in effects such as specularities and subsurface scattering. Please note the view-dependent effects such as specularities in the results (nose, forehead. ). Our method can synthesise results even under high-frequency light conditions resulting in shadows, even though the StyleGAN network is trained on a dataset of natural images.
Ablation Study In this section, we evaluate the importance of the different loss terms of our objective function (Eq. 1). Results are shown in Fig. 7. The target illumination and viewpoint are visualised using a reference image (second column) with the same scene parameters. Removing the latent loss leads to clear distortions of the head geometry. Only using the perceptual loss leads to results with closed eye expressions, as our training data only consists of people captured with closed eyes. We found that the latent loss term helps in generalisation to unseen expressions. However, using only the latent loss is not sufficient for high-quality results. In such case, the facial identity and facial hair (see row 1) are not well preserved, and the relighting is not very accurate (see rows 1,2,6).
Comparisons to Related Methods We compare our method with several state of the art portrait editing approaches. We evaluate qualitatively on in the wild data, as well as quantitatively on the test set of the light-stage data. We compare with the following approaches: • The relighting approach of  Sun et al. [2019]  which is a datadriven technique trained on a light-stage dataset. It can only edit the scene illumination. • The relighting approach of  Zhou et al. [2019]  which is trained on synthetic data. It can also only edit the scene illumination. • PIE  [Tewari et al. 2020b ] is a method which computes a Style-GAN embedding used to edit the image. It can edit the head pose and scene illumination sequentially (unlike ours, which can perform the edits simultaneously). It is trained without supervised image pairs.
We also quantitatively evaluate these results in Tables  1  and 2. In both tables, our method outperforms all related approaches using the Si-MSE metric, even when trained with just 3 identities. All versions of our approach perform similar in terms of SSIM. These evaluations show that while larger datasets lead to better results, only limited supervised data is required to outperform the state of the art. Finally, despite the limited expressivity of the training dataset (subjects in a single expression with eyes and mouth closed), our method is able to generalise to different expressions, as shown in our results (mouth and eyes open, smiling, etc. ) (see Fig. 3 ).
Relative Right Asc [arcsec] After ALMA standard pipeline calibration, a series of preparation steps have to be applied to combine and selfcalibrate the datasets. Using CASA 5. 6. 2, we extract the dust continuum emission by flagging the channels located at ±25 km s -1 from each targeted spectral line. The remaining channels from all spectral windows are averaged into 125 MHz channels. To avoid differences in the weight calculation through different ALMA cycles, we apply the task statwt to recalculate the visibility weighting according to the observed scatter. Next, we aligned the datasets and compared the flux calibration of each individual execution to ensure they were consistent. In Band 7, we found and corrected a 10% discrepancy between the observation from project 2015. 1. 00934. S and all the other observations, and so we rescaled this compact configuration data to match the others.
We used the central frequency of each binned channel to convert the uvcoordinates into wavelength units. All the dust continuum calibration steps, including the centroid shifting, flux calibration, and self-calibration tables, were then applied to the molecular line emission channels. The continuum emission was subtracted using the uvcontsub task. We generate images of 0. 5 km s -1 in velocity width, with a robust parameter of 1. To enhance the S/N, we chose to apply a uv-tapering with a Gaussian of 0. 03 on the 12 CO, and 0. 04 on the 13 CO. We used the package bettermoments  (Teague & Foreman-Mackey 2018)  to create the kinematic map. This package fits a quadratic function to find the peak intensity of the line emission in each pixel, and the velocity associated to it.
Images, dust disk mass, and spectral index Figure  1  shows the final images of CIDA 1 of the ALMA observations in Band 7 (0. 9 mm) and in Band 4 (2. 1 mm) and the details of the images are summarized in Table  2. The fluxes from this table are obtained from the images as-Table  2. Summary of the ALMA images in Fig. 1. Band λ F peak F total σ beam [mm] [Jy beam -1 ] [Jy] [Jy beam -1 ] [mas×mas] 7 0. 9 7. 9 × 10 -4 3. 1 × 10 -2 1. 7 × 10 -5 54×34 4 2. 1 1. 4 × 10 -4 5. 1 × 10 -3 5. 7 × 10 -6 50×34 suming all the emission within a circular area of 0. 5" from the center. The images of the 12 CO and 13 CO are shown in Figure  2 ; these were obtained by processing the channel maps with the Python package bettermoments (Teague & Foreman-Mackey 2018), and clipping the emission at 3σ. The emission of CS was not detected in Band 4, although an increase in the noise level is observed close to the central velocity channel.
Dust morphology from visibility fitting We describe the emission on both ALMA bands with axisymmetric parametric models, and for comparison we also use the Gaussian Processes modeler Python package frank  (Jennings et al. 2020). While a single centrally peaked Gaussian is enough to describe the inner emission of CIDA 1, for the ring emission we tried several different combinations of Gaussian profiles and/or radially broken Gaussian profiles (Gaussians with different widths to each side of the peak). After these tests, we find that the best para-metric models for the ALMA Band 4 (2. 1mm) and Band 7 (0. 9mm) emission are given by: I 2. 1mm (r) = 10 f0 g(r, σ 0 ) + 10 f1 g bg (r -r 1 , σ 1i , σ 1o ), (1) 4. 9 ± 0. 1 29. 2 ± 0. 1 mJy where g is a Gaussian profile, and g bg is a broken Gaussian, described by: I 0.
9mm (r) = 10 f0 g(r, σ 0 ) + 10 f1 g bg (r -r 1 , σ 1i , σ 1o ) +10 f2 g(r, σ 2 ), (2) g(x, σ) = exp - x 2 2 σ 2 , (3) g bg (x, σ) = g(x, σ i ) for x ≤ 0. g(x, σ o ) for x > 0. (4) For each model, the visibilities are obtained by optimizing the model profile with a spatial offset (δ RA , δ Dec ), an inclination and PA, which are used to deproject the observational data. Therefore, each model has four extra free parameters in addition to those that describe the intensity profile. The Fourier Transforms to obtain the models visibilities and the χ 2 calculation are computed with the galario Python package  (Tazzari et al. 2018 ) using a pixel size of 0. 2 mas. We sample the posterior probability distribution with a Markov chain Monte Carlo (MCMC) routine based on the emcee Python package  (Foreman-Mackey et al.
We used the dust evolution package DustPy  (Stammer & Birnstiel in prep). The simulations include the growth, fragmentation, and erosion of particles. The fragmentation velocity of the particles is 10 m s -1 in the entire disk. We assumed α visc = 10 -3 and a disk mass of 5. 5 M Jup around a stellar object of 0. 1 M mass (i. e. , M disk /M CIDA1 ∼ 0. 05). For more details of the dust evolution models see  Birnstiel et al. (2010). Figure  8  shows the dust density distribution as a function of grain size and distance from the star for the case of a ∼ 2. 4 M Jup located at 8 au and three different times of evolution (0. 1, 0. 5, and 1 Myr). The white line corresponds to the particle size for which the Stokes number (St) is equal to unity (St = aρs Σg π 2 ), and therefore it also represents the shape of the gas surface density.
Acknowledgements P. P. and N. T. K. acknowledge support provided by the  Alexander von Humboldt Foundation  in the framework of the  Sofja Kovalevskaja Award  endowed by the  Federal Ministry of Education and Research. This work was partly supported by the  Deutsche Forschungsgemeinschaft (DFG, German Research Foundation ) -Ref no. FOR 2634/1 TE 1024/1-1. M. T. has been supported by the  UK Science and Technology research Council (STFC)  via the consolidated grant  ST/S000623/1 , and by the  European Union 's  Horizon 2020 research and innovation programme  under the  Marie Sklodowska-Curie  grant agreement No. 823823  ( RISE DUSTBUSTERS  project). S. M. S. acknowledges funding from the  European Research Council (ERC)  under the European Unions  Horizon 2020 research and innovation programme  under grant agreement No.
This open-source framework, called MicroAI, is designed as an alternative to existing inference engines (TensorFlow Lite for Microcontrollers and STM32Cube. AI). Our framework can easily be adjusted and/or extended for specific use cases. Executions using single-precision 32-bit floatingpoint as well as fixed-point on 8-and 16-bit integers are supported. The proposed quantization method is evaluated with three different datasets (UCI-HAR, Spoken MNIST and GTSRB). Finally, a comparison study between MicroAI and both existing embedded inference engines is provided in terms of memory and power efficiency. On-device evaluation was done using ARM Cortex-M4F-based microcontrollers (Ambiq Apollo3 and STM32L452RE).
This is of interest because integer operations require considerably fewer computations on most processor cores, including microcontrollers. Without a floating-point unit (FPU), floatingpoint instructions must be emulated in software, creating a large overhead, as was illustrated in  [7]. In that study, a comparison between software, hardware and custom hybrid FPU implementations was provided. In this paper, we present an open-source  [8]  framework, called MicroAI, to perform end-to-end training, quantization and deployment of deep neural networks on microcontrollers. The training phase relies on the well-known TensorFlow and PyTorch deep learning frameworks. Our objective is to provide a framework that is easy to adapt and extend, while maintaining a good compromise between accuracy, energy efficiency and memory footprint.
As a second contribution, we provide some comparative results using two different microcontrollers (STM32L452RE and Ambiq Apollo3) and three different inference engines (TensorFlow Lite for Microcontrollers, STM32Cube. AI and our own MicroAI). Results are compared in terms of memory footprint, inference time and power efficiency. Finally, we propose to apply 8-bit and 16-bit quantization methods on three datasets dealing with different modalities: acceleration and angular velocity from body-worn sensors for UCI-HAR, speech for Spoken MNIST and images for GTSRB. These datasets are light enough to be handled by a deep neural network running on a microcontroller, but still relevant for applications relying on embedded artificial intelligence. Section 2 presents the challenges of running deep neural networks on microcontrollers and recent advances in this field.
Section 3 describes the two common ways of representing real numbers on modern computers: floating point and fixed point. Section 4 presents the methodology implemented in our MicroAI framework for deep neural network quantization. Section 5 details our MicroAI framework and compares it to existing solutions. In Section 6, some comparative results between our framework MicroAI and two popular embedded neural network frameworks (TensorFlow Lite for Microcontrollers and STM32Cube. AI) are given for two microcontroller platforms (Nucleo-L452RE-P and SparkFun Edge) in terms of inference time and power efficiency. The impact of our 8-and 16-bit quantization methods for three different datasets (UCI-HAR, Spoken MNIST and GTSRB) is also presented. In Section 7 the results obtained are discussed. Finally, Section 8 concludes this work and discusses future perspectives.
The quantization step is determined by minimizing the quantization error at the output of the layer, thus balancing the precision and the dynamic range. As the implementation relies on bit shifts rather than on integer multiplications, this solution has some benefits in terms of resource usage and latency for an FPGA target. Additionally, results show that there is a slight degradation of accuracy when using the proposed non-uniform quantization versus a uniform quantization. Using lower-precision computation for deep neural networks has been explored in  [30]  where the authors compare the test error rates on various image datasets for single-precision floating point, half-precision floating point, 20-bit fixed point and their own dynamic fixedpoint approach with 10 bits for activations and 12 bits for weights.