dataesr/scientific-paragraphs-categorization huggingface dataset:
Learning the CPTs for a given network structure is straightforward: with suitably chosen Bayesian priors (a Dirichlet distribution), it is a matter of counting co-occurrences of value pairs in a dataset  [SDLC93]. Learning the structure of a BBN is much more complex: in fact, the general case is NPhard  [Chi96]. Some special cases (such as trees) are tractable, but in this domain it is preferable to fix the structure based on our understanding of the users' analytical process. Models for this process, such as CRISP-DM  [WH00]  (used in data mining) or those drawn from the infovis community (such as the semantic interaction pipeline), are currently rather high-level, and a more detailed task analysis is necessary before the requisite level of detail for a full computational model can be achieved.
It is clear from the discussion throughout this paper that there are barriers to the closer integration of machine learning and visual analytics. One of the main technical barriers is that the current software tools are strongly divided between the research communities. Visualization tools are strong at close control over the form and layout of information, and user interaction: Some tend to be written as bespoke integrated tools, such as Tableau (http://www. tableau. com), Orange (orange. biolab. si) and JMP (www. jpm. com). On the other hand, the most advanced machine-learning tools are often written as libraries in numerical or statistical languages (such as Matlab, e. g. [Nab02] and R), as well, as in high level general purpose languages, like Java (with Weka, a widely used collection of ML algorithms for data mining tasks, or the Stanford NLP tools with advanced ML algorithms for natural language processing) or Python (with powerful and widely adopted data analysis and ML libraries like scipy, scikit-learn, pandas, etc.
); all of them focus on supporting the (often) challenging task of learning complex models from data but provide limited graphical display and interaction. The best solution to this problem, short of reimplementing large toolkits in other languages is to take a client-server approach: a backend server running a good mathematical package for the machine-learning components complemented by web services and html+js clients, able to take advantage of the huge and growing spectrum of javascript libraries and frameworks (such as d3js) to provide interactive information visualisation.
Acknowledgments The authors would like to acknowledge that much of the content and inspiration for this paper originated during a Dagstuhl Seminar titled, "Bridging Machine Learning with Information Visualization (  15101  Cagatay Turkay is a Lecturer (Assistant Prof. ) at the Department of Computer Science at City, University of London. He carries out his research at the giCentre and develops methods where interactive visualisations and computational tools are used in tandem for informed analysis processes. William Wong is a professor of Human-Computer Interaction at Middlesex University, London. He is the head of the Interaction Design Centre. His research interest include investigating the problems of visual analytics in sense-making domains with high information density, such as intelligence analysis, financial systemic risk analysis, and low literacy users.
)" [KMRV15]. In  addition, funding for the authors was provided in part by the Analysis in Motion Initiative at PNNL, Spanish Ministry of Economy & Competitivity and FEDER funds, under grant DPI2015-69891-C2-2-R 9. Author Bios Alex Endert is an Assistant Professor in the School of Interactive Computing at Georgia Tech, where he directs the Visual Analytics Lab. In 2013, his work on Semantic Interaction was awarded the IEEE VGTC VPG Pioneers submitted to COMPUTER GRAPHICS Forum (2/2018). Group Doctoral Dissertation Award, and the Virginia Tech CS Best Dissertation Award. William Ribarsky is the Bank of America Endowed Chair and Director of the Charlotte Visualization Center at UNC Charlotte. He is one of the founders of the field of visual analytics and was Chair of the IEEE Visualization Analytics Science and Technology (VAST) Steering Committee until October, 2015.
PoS(ICRC2017)771 In the recent years, physics experiments have definitely entered the Big Data era. Future telescopes (such as the Square Kilometre Array, the Cherenkov Telescope Array, the Large Synoptic Survey Telescope, Virgo/LIGO. ) will generate more data than ever. This is an exciting time as the analysis and the combination of all these data will lead to new discoveries and even to new ways to do science, but it comes with a price. Data management is now a challenge on its own. The H2020 project ASTERICS is addressing these challenges by bringing together the whole astrophysics and Astroparticle around the ESFRI projects to enable them to interoperate as an integrated, multi-wavelength and multi-messenger telescope. In particular, the OBELICS work package is developing common solutions for the generation, the integration and the analysis of data.
These parameters are at the core of most of the IACT data analysis algorithms. In the last section, we propose an idea for a new reconstruction method based on spectral value decomposition. These values present the advantage to contain more information on the images than the Hillas parameters, thus allowing for more complex analysis including more physics. The developments presented here has been realized in the framework of the ASTERICS-H2020 project and as such are available as an open-source software package to the Astronomy, Astrophysics and Astroparticle communities.
By subtracting v min to the data, b becomes its basis. Once this basis and the data minimum value stored, the data can be stored as contiguous basis blocks, with several blocks (at most d/b) in one unsigned int (see figure  1 ). The complete presentation of the compression algorithm is done in another article (submitted). As an example, an IACT simulation file such as the ones used in CTA (Simtel_array) of 1. 2Go can be compressed with a ratio of about 3. 75 in only 3. 7 seconds whereas the use of the LZMA algorithm results in a compression ratio of 4. 8 in more than 7 minutes. In the second line, the orange block represent the changes between the different values to compress. The last line shows the compressed data (as they are stored). First, the minimum value of the data, next, the basis b = maxmin + 1, and finally the data stored as contiguous blocks.
Moreover, the classical template method are not scalable to the new IACT data volume and structure. The pixel per pixel comparison must be replaced by a lighter method. The Singular Value Decomposition is one of these methods. The Singular Value Decomposition algorithm (SVD) generalized the Fourier transform on matrices. The matrix SVD is : M = UDV T (4. 1) Where M ∈ M n,m (R), U ∈ O n,m (R) (orthogonal matrices n × m), V ∈ O m (R), D ∈ D m (R) (diagonal matrix). The diagonal matrix D is also called singular value spectrum. This spectrum condenses the matrix M information as the frequency spectrum in a Fourier transform. Therefore, the singular value spectrum can be compared as the pixel per pixel comparison to get the primary particle physical characteristics. This method used less data because only few singular values (∼ 7) are relevant to describe the signal in the camera.
Conclusion With the development of new science infrastructures generating unprecedented volumes of data, new HPC approaches are necessary in order to manage and analysis scientific data. In this work we present generic solutions including a new lossless compression algorithm for digitized data, a HPC library providing low-level algorithms and a new approach for template analysis. These developments have been tested and optimised for IACT data. They are available to download 1  as open-source algorithms and can therefore be used by the whole community. We acknowledge support from the ASTERICS project supported by the European Commission Framework Programme Horizon 2020. Research and Innovation action under grant agreement n. 653477.
We propose Aether, an embedded domain specic language for Monte Carlo integration, which oers primitives for writing concise and correct-by-construction sampling and probability code. The user is tasked with writing sampling code, while our compiler automatically generates the code necessary for evaluating PDFs as well as the book keeping and combination of multiple sampling strategies. Our language focuses on ease of implementation for rapid exploration, at the cost of run time performance. We demonstrate the eectiveness of the language by implementing several challenging rendering algorithms as well as a new algorithm, which would otherwise be prohibitively dicult.
INTRODUCTION Probabilistic integration techniques used in lighting simulation shine by their combination of elegance and eciency. The pseudocode for an algorithm like bidirectional path tracing with multiple importance sampling ts in a small gure and reveals at once its power and sophistication. Unfortunately, while the implementation of a simple path tracer is relatively straightforward, achieving a correct implementation of more advanced algorithms, such as bidirectional path tracing or Metropolis light transport, is a major undertaking prone to subtle probability bugs that are extremely challenging to chase down. This is obvious from the very small number of available implementations, and even the widely used pbrt  [Pharr and Humphreys 2010]  did not include full bidirectional path tracing with multiple importance sampling until ten years after the rst release, and a publicly available implementation of Metropolis light transport  [Veach and Guibas 1997]  did not appear until a decade after the original paper [Jakob 2010].
For concreteness, the interested reader can compare and contrast the implementations of path tracing and bidirectional path tracing in either the publicly available Mitsuba or pbrt-v3 renderers. It is readily apparent that the move from path tracing to bidirectional path tracing necessitates large changes in software architecture, including the data structures used for holding path data, as well as a signicant increase in code complexity stemming from computation of multiple importance sampling weights. Now imagine experimenting with new sampling ideas, such as a tridirectional approach that would trace sub-paths from not only the eye and the light source, but also from a known important opening such as a keyhole or a pinhole. This would require the proper probability computation and book keeping for many dierent subpath generation strategies, a daunting task with current pen-and-paper approaches.
To address these diculties, we propose Aether, a new domain specic language that dramatically simplies the implementation of unbiased Monte Carlo integration. Our central goal is to relieve the programmer of the tasks of deriving and implementing probability density functions, performing explicit measure conversions, and dealing with the book keeping and combinatorics of dierent sampling strategies. Importantly, we need to facilitate complex samplers that are able to evaluate their PDFs at arbitrary sample points that may have been sampled through other samplers, as required by, for instance, multiple importance sampling or Metropolis. In our language, the programmer writes only sampling code, and the language automatically generates the necessary density code by computing symbolic derivatives and function inverses at compile time.
In contrast, we know the sampling process and need to compute the corresponding probabilities. We are solving the forward problem, they are solving the reverse problem. Symbolic Algebra Systems. There are numerous existing symbolic algebra systems: Mathematica  [Wolfram Research, Inc. 2016] , SAGE, sympy, and others. While Mathematica and sympy can be incorporated into other programs through compilation or simple importing, none oer a simple way to write compile time expressions in C++. Further, these systems are typically general purpose. In comparison, we only have to handle more restricted operations. This simplies our task of computing symbolic derivatives and inverses since we only have to handle more domain specic scenarios. Dimensional Analysis. SafeGI  [Ou and Pellacini 2010 ] is a C++ software library that oers compile time checking of physical dimensions, units, and geometric spaces for rendering systems.
Sampling and PDF. We provide ve sampling primitives: • Random variables generate one dimensional samples (Section 5. 1) • Random vectors generate multi-dimensional samples (Section 5. 1) • Discrete random variables for making discrete choices (Section 5. 2) • Strategies are a compound sampling primitive. They use random variables and random vectors as building blocks (Section 5. 3) and can also include discrete distributions. They are used to sample the next vertex in a path. • A generic data structure (RandomSequence) for sampling light paths (Section 5. 6). They are constructed from a list of strategies. All PDF computations are handled by our language. In standard renderers, the programmer must manually account for PDFs, which is error prone, and further complicated by the fact some importance sampling code features a mixture of continuous (e.
This method can be called on arbitrary samples, which is enabled by symbolic inversion (Equation  6 ). MIS then becomes simple (Section 5. 4); our language oers a convenience combine() primitive for this purpose. Path Data Structure. Our language introduces the RandomSequence (Section 5. 6) data structure, which we use to store path data. It is the fundamental building block of all algorithm implementations in our language. Random sequences can be extended with new samples, oer the automatically derived Pdf() method for computing the PDF of the path, and have methods for slicing, concatenating, and reversing sequences, while maintaining the same Pdf() interface on the new sequence. These methods are particularly useful in algorithms like bidirectional path tracing where many paths are sliced and concatenated for MIS.
Using the language provided Pdf() method, computing an estimate is simple: To sample cosHemisphere the user provides 2 numbers between 0 and 1 to the Sample() method. These uniforms can be obtained from any random number generator, or they can come from a quasi-Monte Carlo sequence. This estimator is correct by construction because Aether uses symbolic dierentiation to ensure that the sampling and PDF code are consistent. For convenience, Aether provides an Estimator object construct, which stores the integrand and handles the computation of f (x ) p (x ) (and also handles boundary cases, e. g. when p(x ) = 0). It is especially useful when combining multiple samples, where it also handles computing the necessary weighting values.
Sampling Strategies So far, we have seen simple continuous and discrete random variables. We now introduce strategies, the most general sampling construct in Aether, that encapsulate operations such as BSDF importance sampling, light sampling, lens sampling, etc. Strategies combine together continuous random variables and vectors as well as discrete random variables -for instance, for choosing light sources or BSDF layers -while still providing automatic PDF derivation. Strategies are designed to be supplied to random sequences (Section 5. 6) to create incrementally sampled paths. To implement a strategy, the user writes sampling code as usual, but it is encapsulated within a function object with a particular form: The body of the strategy looks similar to the continuous and discrete sampling code we have already seen, with three exceptions: rst, the sampler is wrapped in a function call operator(); second, discrete random variables (emitterDiscrete) are not sampled directly, but are passed as an argument to context.
Random Sequences The random vector introduced above (Section 5. 1) has a xed size and is designed for situations when all its coordinates are sampled at once. Aether oers another important data structure, RandomSequence, which supports the creation of incremental sequences of generic random variables, where each element is assumed to depend only on the previous element. Each element of a random sequence is sampled from a strategy. We use this type to represent transport paths. The type of data stored in the sequence is user provided, e. g. , a Vertex type representing a point on a surface. Consider sampling a 2-vertex path segment starting from an emitter. This is implemented by a random sequence of two strategies, SamplePointOnLightStrategy (dened above), and SampleHemiAndIntersectStrategy, a strategy that picks a cosineweighted direction, traces a ray starting at the previous path vertex, and computes the intersection: The fundamental operation of a random sequence is to Append new elements to the sequence.
IMPLEMENTATION 6. 1 Basic Data Types The basic data types of Aether are oating point numbers, uniforms, random variables, random vectors, and random sequences. Uniform Random Variables. The fundamental operation of Aether is to transform uniform random variables (uniforms) into more general random variables. Uniforms are the only programming variables in the language. They are declared with a unique ID (unique within an expression): variable<1> u1; variable<2> u2; Each uniform declared in this manner instantiates a new type. Expressions. Basic expressions of Aether are composed of literals, uniforms, and constant parameters. The language includes standard mathematical functions (sqrt, sin, cos, tan, etc. ) and operators (*, +, , \) for transforming uniform random variables into more complex expressions.
). They are simplied in the same manner as basic expressions. Vector expressions can also be combined into matrix expressions, which are represented symbolically as typelists of vector expressions. Branching. Aether also provides condition expressions (e. g. 2_l * u1 > 1_l) and logic operators (&&, ||, !). These are used in the language's pattern construct for creating symbolic branch expressions (like if/else, but with a mandatory default case), which is a list of (condition expression, value expression) pairs. For example: Like simpler expressions, these too are simplied automatically at compile time. We use this construct frequently, e. g. for orienting tangent vectors when constructing a coordinate basis at a surface point or branching over dierent possible BSDF samplers (the language provides a CompositeRandomVariable type to simplify the construction of branches over multiple possible samplers).
This is the value of samplePointOnLight. Pdf(p) if the sphericalLight had been discretely sampled. The PdfContext consults its recorded choices again. Both choices have been evaluated; there are no more. The process ends and the nal result is the sum of the two PDFs. The PdfContext maintains its record of discrete choices as a stack so this process works even for multiple discrete random variables in the one strategy. To ensure correctness we require that the strategy is a pure function i. e. the function has no side eects; the result is always the same given the same arguments. Our language assumes this to be the case for all strategies. C++ does not support pure functions so it is the programmer's responsibility to ensure this assumption is true. In particular, care should be taken to avoid any global variables that may introduce side eects.
As a side note, most modern path tracers, including Mitsuba and pbrt, ignore the probability of path termination when computing MIS weights. (Note that this does not break correctness as the weights still sum to one. ) We verify our implementation by comparing to the reference implementation in Mitsuba. Figure  4a  shows a comparison. The scene also showcases the ability of our language to handle dierent types of geometry and layered BRDFs.
Bidirectional Path Tracer The complexity of bidirectional path tracing is a major driving force of our development of the language. The main logic of our Aether implementation (Figure  6 ) is only ten lines longer (after removing all the empty lines and comments) than the previous path tracer. The major additions are the sampling of the emitter subpaths and the extra path slicing and concatenation. In contrast, Mitsuba and pbrt's implementations for bidirectional path tracing are signicantly longer than their unidirectional path tracers. Some variants of bidirectional path tracing perform additional direct light source sampling when concatenating the camera and emitter subpaths. Existing implementation techniques lead to the additional complexity spilling out of the relevant samplers, decreasing readability and maintainability.
Metropolis Light Transport The original Metropolis light transport algorithm proposed by  Veach and Guibas [1997]  is notoriously dicult to implement. To our knowledge, after its introduction in 1997, there was no publicly available implementation until Mitsuba 0. 4 was released in 2012. Our language provides constructs that address both main sources of implementation diculty: the asymmetric Metropolis-Hastings the algorithm creates a virtual point light to light the scene. It is possible to reuse light paths in our language by storing all emitter subpaths in the form of RandomSequence. Gathering the contribution from virtual point lights by slicing and concatenating light paths is then similar to our (bidirectional) path tracing implementation. As in typical implementations of instant radiosity, we also clamp the geometry term at connection to a xed value by rewriting the integrand evaluation function.
Discussion and Limitations In addition to demonstrating that Aether is capable of succinctly expressing a wide range of rendering algorithms, such as Metropolis light transport, gradient-domain path tracing and progressive photon mapping, we found that it enables easy experimentation with dierent sampling schemes. Examples include direct light source importance sampling in the bidirectional path tracer and the tridirectional path tracer. From a software engineering perspective, the consistency between the sampling code and the generated PDF code, and the separation of the sampling and integrand evaluation, means the coupling between the code modules is drastically reduced. The user can easily adjust their sampling algorithms without the need to modify several code modules. Algorithmic Limitations.
Our Metropolis light transport implementation is around 19 times slower than Mitsuba. We found that much of the overhead is due to the re-execution of the sampling strategies using PdfContext and memory allocation/deallocation during PDF computation. We feel, regardless, that providing validation targets for careful implementations of algorithms discovered by the rapid exploration enabled by Aether is a bridge to practical applicability. Still, studying the reasons behind and potentially mitigating the performance issues, e. g. through static analysis and compiler optimization techniques, is an important avenue for future work. Our language, Aether, dramatically reduces the time required for correct implementation of modern unbiased light transport algorithms, as demonstrated by the concise implementations of path tracing, bidirectional path tracing, path space Metropolis light transport, gradient-domain path tracing, instant radiosity, progressive photon mapping, and the novel tridirectional path tracing.
ACKNOWLEDGMENTS Jonathan Ragan-Kelley  provided valuable feedback and helped scope the project from its inception. Matt Pharr  and  Wenzel Jakob  gave us useful insights and encouragement. The door scene was modeled by  Miika Aittala ,  Samuli Laine , and  Jaakko Lehtinen. The Sponza scene was modeled by  Marko Dabrovic. The Cornell box scene with glass lamps was modeled by  Toshiya Hachisuka. This work was partially funded by  DARPA REVEAL and Toyota.