author: L. D. Faddeev
title: Modern Mathematical Physics:
  what it should be

When somebody asks me, what I do in science, I call myself a specialist in mathematical physics. As I have been there for more than 40 years, I have some definite interpretation of this combination of words: "mathematical physics." Cynics or purists can insist that this is neither mathematics nor physics, adding comments with a different degree of malice. Naturally, this calls for an answer, and in this short essay I want to explain briefly my understanding of the subject. It can be considered as my contribution to the discussion about the origin and role of mathematical physics and thus to be relevant for this volume.

The matter is complicated by the fact that the term "mathematical physics" (often abbreviated by MP in what follows) is used in different senses and can have rather different content. This content changes with time, place and person.

I did not study properly the history of science; however, it is my impression that, in the beginning of the twentieth century, the term MP was practically equivalent to the concept of theoretical physics. Not only Henri Poincaré, but also Albert Einstein, were called mathematical physicists. Newly established theoretical chairs were called chairs of mathematical physics. It follows from the documents in the archives of the Nobel Committee that MP had a right to appear both in the nominations and discussion of the candidates for the Nobel Prize in physics . Roughly speaking, the concept of MP covered theoretical papers where mathematical formulae were used.

However, during an unprecedented bloom of theoretical physics in the 20s and 30s, an essential separation of the terms "theoretical" and "mathematical" occurred. For many people, MP was reduced to the important but auxiliary course "Methods of Mathematical Physics" including a set of useful mathematical tools. The monograph of P. Morse and H. Feshbach is a classical example of such a course, addressed to a wide circle of physicists and engineers.

On the other hand, MP in the mathematical interpretation appeared as a theory of partial differential equations and variational calculus. The monographs of R. Courant and D. Hilbert and S. Sobolev are outstanding illustrations of this development. The theorems of existence and uniqueness based on the variational principles, a priori estimates, and imbedding theorems for functional spaces comprise the main content of this direction. As a student of O. Ladyzhenskaya, I was immersed in this subject since the 3rd year of my undergraduate studies at the Physics Department of Leningrad University. My fellow student N. Uraltseva now holds the chair of MP exactly in this sense.

MP in this context has as its source mainly geometry and such parts of classical mechanics as hydrodynamics and elasticity theory. Since the 60s a new impetus to MP in this sense was supplied by Quantum Theory. Here the main apparatus is functional analysis, including the spectral theory of operators in Hilbert space, the mathematical theory of scattering and the theory of Lie groups and their representations. The main subject is the Schrödinger operator. Though the methods and concrete content of this part of MP are essentially different from those of its classical counterpart, the methodological attitude is the same. One sees the quest for the rigorous mathematical theorems about results which are understood by physicists in their own way.

I was born as a scientist exactly in this environment. I graduated from the unique chair of Mathematical Physics, established by V.I. Smirnov at the Physics Department of Leningrad University already in the 30s. In his venture V.I. Smirnov got support from V. Fock, the world famous theoretical physicist with very wide mathematical interests. Originally this chair played the auxiliary role of being responsible for the mathematical courses for physics students. However in 1955 it got permission to supervise its own diploma projects, and I belonged to the very first group of students using this opportunity. As I already mentioned, O.A. Ladyzhenskaya was our main professor. Although her own interests were mostly in nonlinear PDE and hydrodynamics, she decided to direct me to quantum theory. During last two years of undergraduate studies I was to read the monograph of K.O. Friedrichs, "Mathematical Aspects of Quantum Field Theory," and relate it to our group of 5 students and our professor on a special seminar. At the same time my student friends from the chair of Theoretical Physics were absorbed in reading the first monograph on Quantum Electrodynamics by A. Ahieser and V. Berestevsky. The difference in attitudes and language was striking and I was to become accustomed to both.

After my graduation O.A. Ladyzhenskaya remained my tutor but she left me free to choose research topics and literature to read. I read both mathematical papers (i.e. on direct and inverse scattering problems by I.M. Gelfand and B.M. Levitan, V.A. Marchenko, M.G. Krein, A.Ya. Povzner) and "Physical Review" (i.e. on formal scattering theory by M. Gell-Mann, M. Goldberger, J. Schwinger and H. Ekstein) as well. Papers by I. Segal, L. Van-Hove and R. Haag added to my first impressions on Quantum Field Theory taken from K. Friederichs. In the process of this self-education my own understanding of the nature and goals of MP gradually deviated from the prevailing views of the members of the V. Smirnov chair. I decided that it is more challenging to do something which is not known to my colleagues from theoretical physics rather than supply theorems of substantiality. My first work on the inverse scattering problem especially for the many-dimensional Schrödinger operator and that on the three body scattering problem confirm that I really tried to follow this line of thought.

This attitude became even firmer when I began to work on Quantum Field Theory in the middle of the 60s. As a result, my understanding of the goal of MP drastically modified. I consider as the main goal of MP the use of mathematical intuition for the derivation of really new results in the fundamental physics. In this sense, MP and Theoretical Physics are competitors. Their goals in unraveling the laws of the structure of matter coincide. However, the methods and even the estimates of the importance of the results of work may differ quite significally.

Here it is time to say in what sense I use the term "fundamental physics." The adjective "fundamental" has many possible interpretations when applied to the classification of science. In a wider sense it is used to characterize the research directed to unraveling new properties of physical systems. In the narrow sense it is kept only for the search for the basic laws that govern and explain these properties.

Thus, all chemical properties can be derived from the Schrödinger equation for a system of electrons and nuclei. Alternatively, we can say that the fundamental laws of chemistry in a narrow sense are already known. This, of course, does not deprive chemistry of the right to be called a fundamental science in a wide sense.

The same can be said about classical mechanics and the quantum physics of condensed matter. Whereas the largest part of physical research lies now in the latter, it is clear that all its successes including the theory of superconductivity and superfluidity, Bose-Einstein condensation and quantum Hall effect have a fundamental explanation in the nonrelativistic quantum theory of many body systems.

An unfinished physical fundamental problem in a narrow sense is physics of elementary particles. This puts this part of physics into a special position. And it is here where modern MP has the most probable chances for a breakthrough.

Indeed, until recent time, all physics developed along the traditional circle: experiment — theoretical interpretation — new experiment. So the theory traditionally followed the experiment. This imposes a severe censorship on the theoretical work. Any idea, bright as it is, which is not supplied by the experimental knowledge at the time when it appeared is to be considered wrong and as such must be abandoned. Characteristically the role of censors might be played by theoreticians themselves and the great L. Landau and W. Pauli were, as far as I can judge, the most severe ones. And, of course, they had very good reason.

On the other hand, the development of mathematics, which is also to a great extent influenced by applications, has nevertheless its internal logic. Ideas are judged not by their relevance but more by esthetic criteria. The totalitarianism of theoretical physics gives way to a kind of democracy in mathematics and its inherent intuition. And exactly this freedom could be found useful for particle physics. This part of physics traditionally is based on the progress of accelerator techniques. The very high cost and restricted possibilities of the latter soon will become an uncircumventable obstacle to further development. And it is here that mathematical intuition could give an adequate alternative. This was already stressed by famous theoreticians with mathematical inclinations. Indeed, let me cite a paper by P. Dirac from the early 30s:

> The steady progress of physics requires for its theoretical formulation a mathematics that gets continually more advanced. This is only natural and to be expected. What, however, was not expected by the scientific workers of the last century was the particular form that the line of advancement of the mathematics would take, namely, it was expected that the mathematics would get more complicated, but would rest on a permanent basis of axioms and definitions, while actually the modern physical developments have required a mathematics that continually shifts its foundations and gets more abstract. Non-euclidean geometry and non-commutative algebra, which were at one time considered to be purely fictions of the mind and pastimes for logical thinkers, have now been found to be very necessary for the description of general facts of the physical world. It seems likely that this process of increasing abstraction will continue in the future and that advance in physics is to be associated with a continual modification and generalization of the axioms at the base of mathematics rather than with logical development of any one mathematical scheme on a fixed foundation.
>
> There are at present fundamental problems in theoretical physics awaiting solution, *e.g.*, the relativistic formulation of quantum mechanics and the nature of atomic nuclei (to be followed by more difficult ones such as the problem of life), the solution of which problems will presumably require a more drastic revision of our fundamental concepts than any that have gone before. Quite likely these changes will be so great that it will be beyond the power of human intelligence to get the necessary new ideas by direct attempts to formulate the experimental data in mathematical terms. The theoretical worker in the future will therefore have to proceed in a more inderect way. The most powerful method of advance that can be suggested at present is to employ all the resources of pure mathematics in attempts to perfect and generalise the mathematical formalism that forms the existing basis of theoretical physics, and *after* each success in this direction, to try to interpret the new mathematical features in terms of physical entities.

Similar views were expressed by C.N. Yang. I did not find a compact citation, but all spirit of his commentaries to his own collection of papers shows this attitude. Also he used to tell this to me in private discussions.

I believe that the dramatic history of setting the gauge fields as a basic tool in the description of interactions in Quantum Field Theory gives a good illustration of the influence of mathematical intuition on the development of the fundamental physics. Gauge fields, or Yang–Mills fields, were introduced to the wide audience of physicists in 1954 in a short paper by C.N. Yang and R. Mills , dedicated to the generalization of the electromagnetic fields and the corresponding principle of gauge invariance. The geometric sense of this principle for the electromagnetic field was made clear as early as in the late 20s due to the papers of V. Fock and H. Weyl . They underlined the analogy of the gauge (or gradient in the terminology of V. Fock) invariance of the electrodynamics and the equivalence principle of the Einstein theory of gravitation. The gauge group in electrodynamics is commutative and corresponds to the multiplication of the complex field (or wave function) of the electrically charged particle by a phase factor depending on the space–time coordinates. Einstein's theory of gravity provides an example of a much more sophisticated gauge group, namely the group of general coordinate transformation. Both H. Weyl and V. Fock were to use the language of the moving frame with spin connection, associated with local Lorentz rotations. Thus the Lorentz group became the first nonabelian gauge group and one can see in essentially all formulas characteristics of nonabelian gauge fields. However, in contradistinction to the electromagnetic field, the spin connection enters the description of the space-time and not the internal space of electric charge.

In the middle of the 30s, after the discovery of the isotopic spin in nuclear physics, and forming the Yukawa idea of the intermediate boson, O. Klein tried to geometrise these objects. His proposal was based on his 5-dimensional picture. Proton and neutron (as well as electron and neutrino, there were no clear distinction between strong and weak interactions) were put together in an isovector and electromagnetic field and charged vector meson comprised a $2 \times 2$ matrix. However the noncommutative $SU(2)$ gauge group was not mentioned.

Klein's proposal was not received favorably and N. Borh did not recommend him to publish a paper. So the idea remained only in the form of contribution to proceedings of Warsaw Conference "New Theories in Physics" .

The noncommutative group, acting in the internal space of charges, appeared for the first time in the paper of C.N. Yang and R. Mills in 1954. There is no wonder that Yang received a cool reaction when he presented his work at Princeton in 1954. The dramatic account of this event can be found in his commentaries . Pauli was in the audience and immediately raised the question about mass. Indeed the gauge invariance forbids the introduction of mass to the vector charged fields and masslessness leads to the long range interaction, which contradicts the experiment. The only known massless particles (and accompaning long range interactions) are photon and graviton. It is evident from Yang's text, that Pauli was well acquainted with the differential geometry of nonabelian vector fields but his own censorship did not allow him to speak about them. As we know now, the boldness of Yang and his esthetic feeling finally were vindicated. And it can be rightly said, that C.N. Yang proceeded according to mathematical intuition.

In 1954 the paper of Yang and Mills did not move to the forefront of high energy theoretical physics. However, the idea of the charged space with noncommutative symmetry group acquired more and more popularity due to the increasing number of elementary particles and the search for the universal scheme of their classification. And at that time the decisive role in the promotion of the Yang–Mills fields was also played by mathematical intuition.

At the beginning of the 60s, R. Feynman worked on the extension of his own scheme of quantization of the electromagnetic field to the gravitation theory of Einstein. A purely technical difficulty — the abundance of the tensor indices — made his work rather slow. Following the advice of M. Gell-Mann, he exercised first on the simpler case of the Yang–Mills fields. To his surprise, he found that a naive generalization of his diagrammatic rules designed for electrodynamics did not work for the Yang-Mills field. The unitarity of the $S$-matrix was broken. Feynman restored the unitarity in one loop by reconstructing the full scattering amplitude from its imaginary part and found that the result can be interpreted as a subtraction of the contribution of some fictitious particle. However his technique became quite cumbersome beyond one loop. His approach was gradually developed by B. De-Witt . It must be stressed that the physical senselessness of the Yang–Mills field did not preclude Feynman from using it for mathematical construction.

The work of Feynman became one of the starting points for my work in Quantum Field Theory, which I began in the middle of the 60s together with Victor Popov. Another point as important was the mathematical monograph by A. Lichnerowitz , dedicated to the theory of connections in vector bundles. From Lichnerowitz's book it followed clearly that the Yang–Mills field has a definite geometric interpretation: it defines a connection in the vector bundle, the base being the space-time and the fiber the linear space of the representation of the compact group of charges. Thus, the Yang–Mills field finds its natural place among the fields of geometrical origin between the electromagnetic field (which is its particular example for the one-dimensional charge) and Einstein's gravitation field, which deals with the tangent bundle of the Riemannian space-time manifold.

It became clear to me that such a possibility cannot be missed and, notwithstanding the unsolved problem of zero mass, one must actively tackle the problem of the correct quantization of the Yang–Mills field.

The geometric origin of the Yang–Mills field gave a natural way to resolve the difficulties with the diagrammatic rules. The formulation of the quantum theory in terms of Feynman's functional integral happened to be most appropriate from the technical point of view. Indeed, to take into account the gauge equivalence principle one has to integrate over the classes of gauge equivalent fields rather than over every individual configuration. As soon as this idea is understood, the technical realization is rather straightforward. As a result V. Popov and I came out at the end of 1966 with a set of rules valid for all orders of perturbation theory. The fictitious particles appeared as auxiliary variables giving the integral representation for the nontrivial determinant entering the measure over the set of gauge orbits.

Correct diagrammatic rules of quantization of the Yang-Mills field, obtained by V. Popov and me in 1966–1967 , did not attract immediate the attention of physicists. Moreover, the time when our work was done was not favorable for it. Quantum Field Theory was virtually forbidden, especially in the Soviet Union, due to the influence of Landau. "The Hamiltonian is dead" — this phrase from his paper , dedicated to the anniversary of W. Pauli — shows the extreme of Landau's attitude. The reason was quite solid, it was based not on experiment, but on the investigation of the effects of renormalization, which led Landau and his coworkers to believe that the renormalized physical coupling constant is inevitably zero for all possible local interactions. So there was no way for Victor Popov and me to publish an extended article in a major Soviet journal. We opted for the short communication in "Physics Letters" and were happy to be able to publish the full version in the preprint series of newly opened Kiev Institute of Theoretical Physics. This preprint was finally translated into English by B. Lee as a Fermilab preprint in 1972, and from the preface to the translation it follows that it was known in the West already in 1968.

A decisive role in the successful promotion of our diagrammatic rules into physics was played by the works of G. 't Hooft , dedicated to the Yang–Mills field interacting with the Higgs field (and which ultimately led to a Nobel Prize for him in 1999) and the discovery of dimensional transmutation (the term of S. Coleman ). The problem of mass was solved in the first case via the spontaneous symmetry breaking. The second development was based on asymptotic freedom. There exists a vast literature dedicated to the history of this dramatic development. I refer to the recent papers of G. 't Hooft and D. Gross , where the participants in this story share their impressions of this progress. As a result, the Standard Model of unified interactions got its main technical tool. From the middle of the 70s until our time it remains the fundamental base of high energy physics. For our discourse it is important to stress once again that the paper based on mathematical intuition preceded the works made in the traditions of theoretical physics.

The Standard Model did not complete the development of fundamental physics in spite of its unexpected and astonishing experimental success. The gravitational interactions, whose geometrical interpretation is slightly different from that of the Yang–Mills theory, is not included in the Standard Model. The unification of quantum principles, Lorentz–Einstein relativity and Einstein gravity has not yet been accomplished. We have every reason to conjecture that the modern MP and its mode of working will play the decisive role in the quest for such a unification.

Indeed, the new generation of theoreticians in high energy physics have received an incomparably higher mathematical education. They are not subject to the pressure of old authorities maintaining the purity of physical thinking and/or terminology. Futhermore, many professional mathematicians, tempted by the beauty of the methods used by physicists, moved to the position of the modern mathematical physics. Let use cite from the manifesto, written by P. MacPherson during the organization of the Quantum Field Theory year at the School of Mathematics of the Institute for Advanced Study at Princeton:

> The goal is to create and convey an understanding, in terms congenial to mathematicians, of some fundamental notions of physics, such as quantum field theory. The emphasis will be on developing the intuition stemming from functional integrals.
>
> One way to define the goals of the program is by negation, excluding certain important subjects commonly pursued by mathematicians whose work is motivated by physics. In this spirit, it is not planned to treat except peripherally the magnificient new applications of field theory, such as Seiberg-Witten equations to Donaldson theory. Nor is the plan to consider fundamental new constructions within mathimatics that were inspired by physics, such as quantum groups or vertex operator algebras. Nor is the aim to discuss how to provide mathematical rigor for physical theories. Rather, the goal is to develop the sort of intuition common among physicists for those who are used to thought processes stemming from geometry and algebra.

I propose to call the intuition to which MacPherson refers that of mathematical physics. I also recommend the reader to look at the instructive drawing by P. Dijkgraaf on the dust cover of the volumes of lectures given at the School .

The union of these two groups constitutes an enormous intellectual force. In the next century we will learn if this force is capable of substituting for the traditional experimental base of the development of fundamental physics and pertinent physical intuition.


abstract: This article describes the motivation, design, and progress of the Journal of Open Source Software (*JOSS*). *JOSS* is a free and open-access journal that publishes articles describing research software. It has the dual goals of improving the quality of the software submitted and providing a mechanism for research software developers to receive credit. While designed to work within the current merit system of science, *JOSS* addresses the dearth of rewards for key contributions to science made in the form of software. *JOSS* publishes articles that encapsulate scholarship contained in the software itself, and its rigorous peer review targets the software components: functionality, documentation, tests, continuous integration, and the license. A *JOSS* article contains an abstract describing the purpose and functionality of the software, references, and a link to the software archive. The article is the entry point of a *JOSS* submission, which encompasses the full set of software artifacts. Submission and review proceed in the open, on GitHub. Editors, reviewers, and authors work collaboratively and openly. Unlike other journals, *JOSS* does not reject articles requiring major revision; while not yet accepted, articles remain visible and under review until the authors make adequate changes (or withdraw, if unable to meet requirements). Once an article is accepted, *JOSS* gives it a digital object identifier (DOI), deposits its metadata in Crossref, and the article can begin collecting citations on indexers like Google Scholar and other services. Authors retain copyright of their *JOSS* article, releasing it under a Creative Commons Attribution 4.0 International License. In its first year, starting in May 2016, *JOSS* published 111 articles, with more than 40 additional articles under review. *JOSS* is a sponsored project of the nonprofit organization NumFOCUS and is an affiliate of the Open Source Initiative (OSI).
author: Arfon M. Smith[^1]; Kyle E. Niemeyer; Daniel S. Katz; Lorena A. Barba; George Githinji; Melissa Gymrek; Kathryn D. Huff; Christopher R. Madan; Abigail Cabunoc Mayes; Kevin M. Moerman; Pjotr Prins; Karthik Ram; Ariel Rokem; Tracy K. Teal; Roman Valls Guimera; Jacob T. Vanderplas
bibliography: references.bib
date: January 2018
title: Journal of Open Source Software (JOSS): design and first-year review

# Introduction

Modern scientific research produces many outputs beyond traditional articles and books. Among these, research software is critically important for a broad spectrum of fields. Current practices for publishing and citation do not, however, acknowledge software as a first-class research output. This deficiency means that researchers who develop software face critical career barriers. The *Journal of Open Source Software* (*JOSS*) was founded in May 2016 to offer a solution within the existing publishing mechanisms of science. It is a developer-friendly, free and open-access, peer-reviewed journal for research software packages. By its first anniversary, *JOSS* had published more than a hundred articles. This article discusses the motivation for creating a new software journal, delineates the editorial and review process, and summarizes the journal's first year of operation via submission statistics. We expect this article to be of interest to three core audiences: (1) researchers who develop software and could submit their work to *JOSS*, (2) those in the community with an interest in advancing scholarly communications who may appreciate the technical details of the *JOSS* journal framework, and (3) those interested in possibilities for citing software in their own research publications.

The sixteen authors of this article are the members of the *JOSS* Editorial Board at the end of its first year (May 2017). Arfon Smith is the founding editor-in-chief, and the founding editors are Lorena A. Barba, Kathryn Huff, Daniel Katz, Christopher Madan, Abigail Cabunoc Mayes, Kevin Moerman, Kyle Niemeyer, Karthik Ram, Tracy Teal, and Jake Vanderplas. Five new editors joined in the first year to handle areas not well covered by the original editors, and to help manage the large and growing number of submissions. They are George Githinji, Melissa Gymrek, Pjotr Prins, Ariel Rokem, and Roman Valls Guimera. (Since then, we added three more editors: Jason Clark, Lindsey Heagy, and Thomas Leeper.)

The *JOSS* editors are firm supporters of open-source software for research, with extensive knowledge of the practices and ethics of open source. This knowledge is reflected in the *JOSS* submission system, peer-review process, and infrastructure. The journal offers a familiar environment for developers and authors to interact with reviewers and editors, leading to a citable published work: a software article. The article describes the software at a high level, and the software itself includes both source code and associated artifacts such as tests, documentation, and examples. With a Crossref digital object identifier (DOI), the article is able to collect citations, empowering the developers/authors to gain career credit for their work. *JOSS* thus fills a pressing need for computational researchers to advance professionally, while promoting higher quality software for science. *JOSS* also supports the broader open-science movement by encouraging researchers to share their software openly and follow best practices in its development.

# Background and motivation

A 2014 study of UK Russell Group Universities  reports that $\sim$`<!-- -->`{=html}90% of academics surveyed said they use software in their research, while more than 70% said their research would be impractical without it. About half of these UK academics said they develop their own software while in the course of doing research. Similarly, a 2017 survey of members of the US National Postdoctoral Association found that 95% used research software, and 63% said their research would be impractical without it .

Despite being a critical part of modern research, software lacks support across the scholarly ecosystem for its publication, acknowledgement, and citation . Academic publishing has not changed substantially since its inception. Science, engineering, and many other academic fields still view research articles as the key indicator of research productivity, with research grants being another important indicator. Yet, the research article is inadequate to fully describe modern, data-intensive, computational research. *JOSS* focuses on research software and its place in the scholarly publishing ecosystem.

## Why publish software?

Most academic fields still rely on a one-dimensional credit model where academic articles and their associated citations are the dominant factor in the success of a researcher's career. Software creators, in order to increase the likelihood of receiving career credit for their work, often choose to publish "software articles" that act as placeholder publications pointing to their software. At the same time, recent years have seen a push for sharing open research software .

Beyond career-credit arguments for software creators, publishing research software enriches the scholarly record. Buckheit and Donoho paraphrased Jon Claerbout, a pioneer of reproducible research, as saying: "An article about a computational result is advertising, not scholarship. The actual scholarship is the full software environment, code and data, that produced the result." . The argument that articles about computational science are not satisfactory descriptions of the work, needing to be supplemented by code and data, is more than twenty years old! Yet, despite the significance of software in modern research, documenting its use and including it in the scholarly ecosystem presents numerous challenges.

## Challenges of publishing software

The conventional publishing mechanism of science is the research article, and a researcher's career progression hinges on collecting citations for published works. Unfortunately, software citation  is in its infancy (as is data citation ). Publishing the software itself and receiving citation credit for it may be a better long-term solution, but this is still impractical. Even when software (and data) are published so that they can be cited, we do not have a standard culture of peer review for them. This leads many developers today to publish software articles.

The developer's next dilemma is where to publish, given the research content, novelty, length and other features of a software article. Since 2012, Neil Chue Hong has maintained a growing list of journals that accept software articles . He includes both generalist journals, accepting software articles from a variety of fields, and domain-specific journals, accepting both research and software articles in a given field. For many journals, particularly the domain-specific ones, a software article must include novel results to justify publication.

From the developer's point of view, writing a software article can involve a great deal of extra work. Good software includes documentation for both users and developers that is sufficient to make it understandable. A software article may contain much of the same content, merely in a different format, and developers may not find value in rewriting their documentation in a manner less useful to their users and collaborators. These issues may lead developers to shun the idea of software articles and prefer to publish the software itself. Yet, software citation is not common and the mostly one-dimensional credit model of academia (based on article citations) means that publishing software often does not "count" for career progression .

# The Journal of Open Source Software

To tackle the challenges mentioned above, the *Journal of Open Source Software* (*JOSS*) launched in May 2016  with the goal of drastically reducing the overhead of publishing software articles. *JOSS* offers developers a venue to publish their complete research software wrapped in relatively short high-level articles, thus enabling citation credit for their work. In this section we describe the goals and principles, infrastructure, and business model of *JOSS*, and compare it with other software journals.

## Goals and principles

*JOSS* articles are deliberately short and only include an abstract describing the high-level functionality of the software, a list of the authors of the software (with their affiliations), a list of key references, and a link to the software archive and software repository. Articles are not allowed to include other content often found in software articles, such as descriptions of the API (application programming interface) and novel research results obtained using the software. The software API should already be described in the software documentation, and domain research results do not belong in *JOSS*—these should be published in a domain journal. Unlike most journals, which ease discoverability of new research and findings, *JOSS* serves primarily as a mechanism for software developers/authors to improve and publish their research software. Thus, software discovery is a secondary feature.

The *JOSS* design and implementation are based on the following principles:

-   Other than their short length, *JOSS* articles are conventional articles in every other sense: the journal has an ISSN, articles receive Crossref DOIs with high-quality submission metadata, and articles are appropriately archived.

-   Because software articles are "advertising" and simply pointers to the *actual* scholarship (the software), short abstract-length submissions are sufficient for these "advertisements."

-   Software is a core product of research and therefore the software itself should be archived appropriately when submitted to and reviewed in *JOSS*.

-   Code review, documentation, and contributing guidelines are important for open-source software and should be part of any review. In *JOSS*, they are the focus of peer review. (While a range of other journals publish software, with various peer-review processes, the focus of the review is usually the submitted article and reviewers might not even look at the code.) The *JOSS* review process itself, described in §, was based on the on-boarding checklist for projects joining the rOpenSci collaboration .

Acceptable *JOSS* submissions also need to meet the following criteria:

-   The software must be open source by the Open Source Initiative (OSI) definition ([opensource.org](https://opensource.org)).

-   The software must have a research application.

-   The submitter should be a major contributor to the software they are submitting.

-   The software should be a significant new contribution to the available open-source software that either enables some new research challenge(s) to be addressed or makes addressing research challenges significantly better (e.g., faster, easier, simpler).

-   The software should be feature-complete, i.e., it cannot be a partial solution.

## How *JOSS* works

*JOSS* is designed as a small collection of open-source tools that leverage existing infrastructure such as GitHub, Zenodo, and Figshare. A goal when building the journal was to minimize the development of new tools where possible.

### The *JOSS* web application and submission tool

The *JOSS* web application and submission tool is hosted at <http://joss.theoj.org>. It is a simple Ruby on Rails web application  that lists accepted articles, provides the article submission form (see Figure ), and hosts journal documentation such as author submission guidelines. This application also automatically creates the review issue on GitHub once a submission has been pre-reviewed by an editor and accepted to start peer review in *JOSS*.

### Open peer review on GitHub

*JOSS* conducts reviews on the `joss-reviews` GitHub repository . Review of a submission begins with the opening of a new GitHub issue, where the editor-in-chief assigns an editor, the editor assigns a reviewer, and interactions between authors, reviewer(s), and editor proceed in the open. Figure  shows an example of a recent review for the (accepted) `hdbscan` package . The actual review includes the code, software functionality/performance claims, test suite (if present), documentation, and any other material associated with the software.

### Whedon and the Whedon-API

Many of the tasks associated with *JOSS* reviews and editorial management are automated. A core RubyGem library named `Whedon`  handles common tasks associated with managing the submitted manuscript, such as compiling the article (from its Markdown source) and creating Crossref metadata. An automated bot, `Whedon-API` , handles other parts of the review process (such as assigning editors and reviewers based on editor input) and leverages the `Whedon` RubyGem library. For example, to assign the editor for a submission, one may type the following command in a comment box within the GitHub issue: `@whedon assign @danielskatz as editor`. Similarly, to assign a reviewer, one enters: `@whedon assign @zhaozhang as reviewer` (where the reviewer and editor GitHub handles identify them). The next section describes the review process in more detail.

## Business model and content licensing

*JOSS* is designed to run at minimal cost with volunteer labor from editors and reviewers. The following fixed costs are currently incurred:

-   Crossref membership: \$275. This is a yearly fixed cost for the *JOSS* parent entity—*Open Journals*—so that article DOIs can be registered with Crossref.

-   Crossref article DOIs: \$1. This is a fixed cost per article.

-   *JOSS* web application hosting (currently with Heroku): \$19 per month

Assuming a publication rate of 100 articles per year results in a core operating cost of $\sim$\$6 per article. With 200 articles per year—which seems possible for the second year—the cost drops to $\sim$\$3.50 per article: $$\begin{aligned}
\label{costs}
(\$275 + (\$1 \times 100) + (\$19 \times 12)) / 100 &= \$6.03 \\
(\$275 + (\$1 \times 200) + (\$19 \times 12)) / 200 &= \$3.51 \;.
\end{aligned}$$

Submitting authors retain copyright of *JOSS* articles and accepted articles are published under a Creative Commons Attribution 4.0 International License . Any code snippets included in *JOSS* articles are subject to the MIT license  regardless of the license of the submitted software package under review, which itself must be licensed under an OSI-approved license (see [opensource.org/licenses/alphabetical](https://opensource.org/licenses/alphabetical) for a complete list).

## Comparison with other software journals

A good number of journals now accept, review, and publish software articles , which we group into two categories. The first category of journals include those similar to *JOSS*, which do not focus on a specific domain and only consider submissions of software/software articles: the *Journal of Open Research Software* (*JORS*, [openresearchsoftware.metajnl.com](http://openresearchsoftware.metajnl.com)), *SoftwareX* ([journals.elsevier.com/softwarex/](https://www.journals.elsevier.com/softwarex/)), and now *JOSS*. Both *JORS*  and *SoftwareX*  now review both the article text and the software. In *JOSS*, the review process focuses mainly on the software and associated material (e.g., documentation) and less on the article text, which is intended to be a brief description of the software. The role and form of peer review also varies across journals. In *SoftwareX* and *JORS*, the goal of the review is both to decide if the article is acceptable for publication and to improve it iteratively through a non-public, editor-mediated interaction between the authors and the anonymous reviewers. In contrast, *JOSS* has the goal of accepting most articles after improving them as needed, with the reviewers and authors communicating directly and publicly through GitHub issues.

The second category includes domain-specific journals that either accept software articles as a special submission type or exclusively consider software articles targeted at the domain. For example, *Collected Algorithms* (CALGO, [acm.org/calgo](http://www.acm.org/calgo/)) is a long-running venue for reviewing and sharing mathematical algorithms associated with articles published in *Transactions on Mathematical Software* and other ACM journals. However, CALGO authors must transfer copyright to ACM and software is not available under an open-source license—this contrasts with *JOSS*, where authors retain copyright and software must be shared under an open-source license. *Computer Physics Communications* ([journals.elsevier.com/computer-physics-communications](https://www.journals.elsevier.com/computer-physics-communications)) and *Geoscientific Model Development* ([geoscientific-model-development.net](https://www.geoscientific-model-development.net/)) publish full-length articles describing application software in computational physics and geoscience, respectively, where review primarily focuses on the article. Chue Hong maintains a list of journals in both categories .

# Peer review in *JOSS*

In this section, we illustrate the *JOSS* submission and review process using a representative example, document the review criteria provided to authors and reviewers, and explain a fast-track option for already-reviewed rOpenSci contributions.

## The *JOSS* process

Figure  shows a typical *JOSS* submission and review process, described here in more detail using the `hdbscan` package  as an example:

1.  Leland McInnes submitted the `hdbscan` software and article to *JOSS* on 26 February 2017 using the web application and submission tool. The article is a Markdown file named `paper.md`, visibly located in the software repository (here, and in many cases, placed together with auxiliary files in a `paper` directory).

2.  Following a routine check by a *JOSS* administrator, a "pre-review" issue was created in the `joss-reviews` GitHub repository . In this pre-review issue, an editor (Daniel S. Katz) was assigned, who then identified and assigned a suitable reviewer (Zhao Zhang). Editors generally identify one or more reviewers from a pool of volunteers based on provided programming language and/or domain expertise.[^2]

    The editor then asked the automated bot `Whedon` to create the main submission review issue via the command `@whedon start review magic-word=bananas`. ("`magic-word=bananas`" is a safeguard against accidentally creating a review issue prematurely.)

3.  The reviewer then conducted the submission review  (see Figure ) by working through a checklist of review items, as described in §. The author, reviewer, and editor discussed any questions that arose during the review, and once the reviewer completed their checks, they notified the submitting author and editor. Compared with traditional journals, *JOSS* offers the unique feature of holding a discussion—in the open within a GitHub issue—between the reviewer(s), author(s), and editor. Like a true conversation, discussion can go back and forth in minutes or seconds, with all parties contributing at will. This contrasts with traditional journal reviews, where the process is merely an exchange between the reviewer(s) and author(s), via the editor, which can take months for each communication, and in practice is limited to one or two, perhaps three in some cases, exchanges due to that delay .

    Note that *JOSS* reviews are subject to a code of conduct , adopted from the Contributor Covenant Code of Conduct . Both authors and reviewers must confirm that they have read and will adhere to this Code of Conduct, during submission and with their review, respectively.

4.  After the review was complete, the editor asked the submitting author to make a permanent archive of the software (including any changes made during review) with a service such as Zenodo or Figshare, and to post a link to the archive in the review thread. This link, in the form of a DOI, was associated with the submission via the command `@whedon set 10.5281/zenodo.401403 as archive`.

5.  The editor-in-chief used the `Whedon` RubyGem library on his local machine to produce the compiled PDF, update the *JOSS* website, deposit Crossref metadata, and issue a DOI for the submission ([10.21105/joss.00205](https://doi.org/10.21105/joss.00205)).

6.  Finally, the editor-in-chief updated the review issue with the *JOSS* article DOI and closed the review. The submission was then accepted into the journal.

Authors can also first submit a pre-submission inquiry via an issue in the main *JOSS* repository  if they have questions regarding the suitability of their software for publication, or for any other questions.

## *JOSS* review criteria

As previously mentioned, the *JOSS* review is primarily concerned with the material in the software repository, focusing on the software and documentation. We do not ask authors to use their software in a research study or include research results in their article beyond as examples; submissions focused on results rather than software should be submitted to research journals. The specific items in the reviewer checklist are:

-   Conflict of interest

    -   As the reviewer I confirm that I have read the *JOSS* [conflict of interest policy](https://github.com/openjournals/joss/blob/master/COI.md) and that there are no conflicts of interest for me to review this work.

-   Code of Conduct

    -   I confirm that I read and will adhere to the [*JOSS* code of conduct](http://joss.theoj.org/about#code_of_conduct).

-   General checks

    -   **Repository**: Is the source code for this software available at the repository URL?

    -   **License**: Does the repository contain a plain-text LICENSE file with the contents of an OSI-approved software license?

    -   **Version**: Does the release version given match the GitHub release?

    -   **Authorship**: Has the submitting author made major contributions to the software?

-   Functionality

    -   **Installation**: Does installation proceed as outlined in the documentation?

    -   **Functionality**: Have the functional claims of the software been confirmed?

    -   **Performance**: Have any performance claims of the software been confirmed?

-   Documentation

    -   **A statement of need**: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

    -   **Installation instructions**: Is there a clearly-stated list of dependencies? Ideally these should be handled with an automated package management solution.

    -   **Example usage**: Do the authors include examples of how to use the software (ideally to solve real-world analysis problems)?

    -   **Functionality documentation**: Is the core functionality of the software documented to a satisfactory level (e.g., API method documentation)?

    -   **Automated tests**: Are there automated tests or manual steps described so that the function of the software can be verified?

    -   **Community guidelines**: Are there clear guidelines for third parties wishing to 1) contribute to the software, 2) report issues or problems with the software, and 3) seek support?

-   Software paper

    -   **Authors**: Does the `paper.md` file include a list of authors with their affiliations?

    -   **A statement of need**: Do the authors clearly state what problems the software is designed to solve and who the target audience is?

    -   **References**: Do all archival references that should have a DOI list one (e.g., papers, datasets, software)?

## Fast track for reviewed rOpenSci contributions

For submissions of software that has already been reviewed under rOpenSci's rigorous onboarding guidelines , *JOSS* does not perform further review. The editor-in-chief is alerted with a note "This submission has been accepted to rOpenSci. The review thread can be found at `[LINK TO ONBOARDING ISSUE]`," allowing such submissions to be fast-tracked to acceptance.

# A review of the first year

By the end of May 2017, *JOSS* published 111 articles since its inception in May 2016, and had an additional 41 articles under consideration. Figure  shows the monthly and cumulative publication rates; on average, we published 8.5 articles per month, with some (nonstatistical) growth over time.

Figure  shows the numbers of days taken for processing and review of the 111 published articles (i.e., time between submission and publication), including finding a topic editor and reviewer(s). Since the journal's inception in May 2016, articles spent on average 45.5 days between submission and publication (median 32 days, interquartile range 52.3 days) The shortest review took a single day, for `Application Skeleton` , while the longest review took 190 days, for `walkr` . In the former case, the rapid turnaround can be attributed to the relatively minor revisions needed (in addition to quick editor, reviewer, and author actions and responses). In contrast, the latter case took much longer due to delays in selecting an editor and finding an appropriate reviewer, and a multimonth delay between selecting a reviewer and receiving reviews. In other cases with long review periods, some delays in responding to requests for updates may be attributed to reviewers (or editors) missing GitHub notifications from the review issue comments. We have already taken steps to improve the ability of authors, reviewers, and editors to keep track of their submissions, including a prompt to new reviewers to unsubscribe from the main `joss-reviews` repository  (to reduce unnecessary notifications) and a weekly digest email for *JOSS* editors to keep track of their submissions. In the future we may collect the email addresses of reviewers so we can extend this functionality to them.

Figure  shows the frequency of programming languages appearing in *JOSS* articles. Python appears the most with over half of published software articles (54), while R is used in nearly one-third of articles (29). We believe the popularity of Python and R in *JOSS* submissions is the result of (1) the adoption of these languages (and open-source practices) in scientific computing communities and (2) our relationship with the rOpenSci project.

Each article considered by *JOSS* undergoes review by one or more reviewers. The set of 111 published articles have been reviewed by 93 unique reviewers. The majority of articles received a review by one reviewer (average of $1.11\pm 0.34$), with a maximum of three reviewers. Based on available data in the review issues, on average, editors reached out to 1.85$\pm$`<!-- -->`{=html}1.40 potential reviewers (at most 8 in one case) via mentions in the GitHub review issue. This does not include external communication, e.g., via email or Twitter. Overall, *JOSS* editors contacted 1.65 potential reviewers for each actual review (based on means).

Interestingly, the current reviewer list contains only 52 entries, as of this writing . Considering the unique reviewer count of 93, we clearly have reached beyond those that volunteered to review a priori. Benefits of using GitHub's issue infrastructure and our open reviews include: 1) the ability to tag multiple people, via their GitHub handles, to invite them as potential reviewers; 2) the discoverability of the work so that people may volunteer to review without being formally contacted; 3) the ability to get additional, unprompted feedback and comments; and 4) the ability to find reviewers by openly advertising, e.g., on social media. Furthermore, GitHub is a well-known, commonly used platform where many (if not most) potential authors and reviewers already have accounts.

Figure  shows the numbers of articles managed by each of the *JOSS* editors. Editor-in-chief Arfon Smith stewarded the majority of articles published in the first year. This was somewhat unavoidable in the first three months after launch, as Smith served as the de facto sole editor for all submissions, with other members of the editorial board assisting. This strategy was not sustainable and, over time, we adopted the pre-review/review procedure to hand off articles to editors. Also, authors can now select during submission the appropriate editor based on article topic.

Lastly, we analyzed the affiliations of the 286 authors associated with articles published in the first year. Figure  shows the number of authors per country; we represented authors with multiple affiliations in different countries using their first affiliation. Authors with no affiliation, or where we could not identify the country, are shown as "unknown." From the articles published in the first year, approximately 48% of authors live in the United States and approximately 40% live in Europe (including Switzerland). The remaining 12% come from the rest of the world, most notably Australia (6.6%) and Canada (2.1%). Moving forward, we hope to receive submissions from authors in more countries that even better represent who develops research software around the world; one strategy to achieve this involves continuing to expand our editorial board.

In its first year, *JOSS* also developed formal relationships with two US-based nonprofit organizations. In March 2017, *JOSS* became a community affiliate of the Open Source Initiative ([opensource.org](https://opensource.org)), the steward of the open-source definition, which promotes open-source software and educates about appropriate software licenses. And, in April 2017, *JOSS* became a fiscally sponsored project of NumFOCUS ([numfocus.org](https://www.numfocus.org)), a 501(c)(3) charity that supports and promotes "world-class, innovative, open source scientific computing." Being associated with these two prominent community organizations increases the trust of the community in our efforts. Furthermore, as a NumFOCUS project, *JOSS* will be able to raise funding to sustain its activities and grow.

# The second year for *JOSS*

Our focus for the second year will be on continuing to provide a high-quality experience for submitting authors and reviewers, and making the best use of the editorial board. In our first year, we progressed from a model where the editor-in-chief handled most central functions to one with more distributed roles for the editors, particularly that of ensuring that reviews are useful and timely. Editors can now select and self-assign to submissions they want to manage, while the editor-in-chief only assigns the remaining submissions. As *JOSS* grows, the process of distributing functions across the editorial board will continue to evolve—and more editors may be needed.

In the second year, we plan to complete a number of high-priority improvements to the *JOSS* toolchain. Specifically, we plan on automating the final steps for accepting an article. For example, generating Crossref metadata and compiling the article are both currently handled by the editor-in-chief on his local machine using the `Whedon` RubyGem library. In the future, we would like authors and reviewers to be able to ask the `Whedon-API` bot to compile the paper for them, and other editors should be able to ask the bot to complete the submission of Crossref metadata on their behalf. Other improvements are constantly under discussion on the *JOSS* GitHub repository ([github.com/openjournals/joss/issues](https://github.com/openjournals/joss/issues)). In fact, anyone is able to report bugs and suggest enhancements to the experience. And, since the *JOSS* tools are open source, we welcome contributions in the form of bug-fixes or enhancements via the usual pull-request protocols.

Beyond roles and responsibilities for the editors, and improvements to the *JOSS* tools and infrastructure, we will take on the more tricky questions about publishing software, such as how to handle new software versions. Unlike traditional research articles that remain static once published, software usually changes over time, at least for maintenance and to avoid software rot/collapse (where software stops working because of changes in the environment, such as dependencies on libraries or operating system). Furthermore, because all potential uses of the software are not known at the start of a project, the need or opportunity arises to add features, improve performance, improve accuracy, etc. After making one or more changes, software developers frequently update the software with a new version number. Over time, the culmination of these changes may result in a major update to the software, and with many new contributors a new version might correspond to a new set of authors if the software is published. However, this process may not translate clearly to *JOSS*. The editorial board will accept a new *JOSS* article published with each major version or even a minor version if the changes seem significant enough to the editor and reviewer(s), but we do not yet know if this will satisfy the needs of both developers and users (corresponding to *JOSS* authors and readers, respectively).

The discussion about new software versions also generally applies to software forks, where software is copied and, after some divergent development, a new software package emerges. Similar to how we handle new software versions, the *JOSS* editorial board will consider publication of an article describing a forked version of software if it includes substantial changes from a previously published version. Authorship questions may be more challenging when dealing with forks compared with new versions, since forks can retain varying amounts of code from the original projects. However, while a version control history generally makes it easy to suggest people who should be authors, deciding on authorship can be difficult and subjective, and is therefore ultimately project-dependent. We prefer to leave authorship decisions to the projects, with discussion taking place as needed with reviewers and editors.

# Conclusions

Software today encapsulates—and generates—important research knowledge, yet it has not entered the science publication ecosystem in a practical way. This situation is costly for science, through the lack of career progression for valuable personnel: research software developers. We founded *JOSS* in response to the acute need for an answer to this predicament. *JOSS* is a venue for authors who wish to receive constructive peer feedback, publish, and collect citations for their research software. By encouraging researchers to develop their software following best practices, and then share and publish it openly, *JOSS* supports the broader open-science movement. The number of submissions confirms the keen demand for this publishing mechanism: more than 100 accepted articles in the first year and more than 40 others under review. By the end of 2017, *JOSS* has published nearly 200 articles. Community members have also responded positively when asked to review submissions in an open and non-traditional format, contributing useful reviews of the submitted software.

However, we are still overcoming initial hurdles to achieve our goals. *JOSS* is currently not fully indexed by Google Scholar, despite the fact that *JOSS* articles include adequate metadata and that we made an explicit request for inclusion in March 2017 (see GitHub [issue \#130](https://github.com/openjournals/joss/issues/130)). Also, we may need to invest more effort into raising awareness of good practices for citing *JOSS* articles. That said, we have some preliminary citation statistics: according to Google Scholar, `corner.py`  and `Armadillo`  have been cited the most at 116 and 79 times, respectively. Crossref's Cited-by service—which relies on publishers depositing reference information—reports 45 and 28 citations for the same articles . While most other articles have received no citations to-date, a few have been cited between one and five times. We have had at least two "repeat" submissions, i.e., submissions of a new version with major changes from a prior version. Clementi et al.  published `PyGBe-LSPR`, a new version that added substantially new features over the original `PyGBe` of Cooper et al. . Similarly, the software published by Sandersen and Curtin  extended on (and cited) their earlier article .

The journal cemented its position in the first year of operation, building trust within the community of open-source research-software developers and growing in name recognition. It also earned weighty affiliations with OSI and NumFOCUS, the latter bringing the opportunity to raise funding for sustained operations. Although publishing costs are low at \$3–6 per article, *JOSS* does need funding, with the editor-in-chief having borne the expenses personally to pull off the journal launch. Incorporating a small article charge (waived upon request) may be a route to allow authors to contribute to *JOSS* in the future, but we have not yet decided on this change. Under the NumFOCUS nonprofit umbrella, *JOSS* is now eligible to seek grants for sustaining its future, engaging in new efforts like outreach, and improving its infrastructure and tooling.

Outreach to other communities still unaware of *JOSS* is certainly part of our growth strategy. Awareness of the journal so far has mostly spread through word-of-mouth and social networking , plus a couple of news articles . As of August 2017, *JOSS* is also listed in the Directory of Open Access Journals (DOAJ) ([doaj.org/toc/2475-9066](https://doaj.org/toc/2475-9066)). We plan to present *JOSS* at relevant domain conferences, like we did at the 2017 SIAM Conference on Computational Science & Engineering  and the 16th Annual Scientific Computing with Python Conference (SciPy 2017). We are also interested in partnering with other domain journals that focus on (traditional) research articles. In such partnerships, traditional peer review of the research would be paired with peer review of the software, with *JOSS* taking responsibility for the latter.

Finally, the infrastructure and tooling of *JOSS* have unexpected added values: while developed to support and streamline the *JOSS* publication process, these open-source tools generalize to a lightweight journal-management system. The *JOSS* web application and submission tool, the `Whedon` RubyGem library, and the `Whedon-API` bot could be easily forked to create overlay journals for other content types (data sets, posters, figures, etc.). The original artifacts could be archived on other services such as Figshare, Zenodo, Dryad, arXiv, or engrXiv/AgriXiv/LawArXiv/PsyArXiv/SocArXiv/bioRxiv. This presents manifold opportunities to expand the ways we assign career credit to the digital artifacts of research. *JOSS* was born to answer the needs of research software developers to thrive in the current merit traditions of science, but we may have come upon a generalizable formula for digital science.

# Acknowledgements

This work was supported in part by the Alfred P. Sloan Foundation. Work by K. E. Niemeyer was supported in part by the National Science Foundation (No. ACI-1535065). Work by P. Prins was supported by the National Institute of Health (R01 GM123489, 2017–2022). Work by K. Ram was supported in part by The Leona M. and Harry B. Helmsley Charitable Trust (No. 2016PG-BRI004). Work by A. Rokem was supported by the Gordon & Betty Moore Foundation and the Alfred P. Sloan Foundation, and by grants from the Bill & Melinda Gates Foundation, the National Science Foundation (No. 1550224), and the National Institute of Mental Health (No. 1R25MH112480).

[^1]: Corresponding author, <firstname.lastname@example.com>

[^2]: Potential reviewers can volunteer via <http://joss.theoj.org/reviewer-signup.html>


author: Tommaso Urli [^1]
[`email@example.com`](mailto:firstname.lastname@example.com)
bibliography: paper.bib
title: Technical Report
  json2run: a tool for experiment design & analysis

# Introduction

**json2run** is a tool to automate the running, storage and analysis of experiments. It has been created in the first place to study different algorithms or different sets of values for algorithm parameters, but it is a general tool and can be used wherever it fits. The main advantage of **json2run** (over a home-brewed experiment suite) is that it allows to describe a set of experiments concisely as a [JSON](http://www.json.org)-formatted parameter tree, such as the following (note the presence of parameter definitions as well as logical operators to combine them)

An experiment file such as the one above describes the parameters that must be generated and passed over to the executable. We'll call a set of generated parameters a *configuration* or *parameter configuration*. Once the experiments have been described, **json2run** can parse the file and perform various operations with it, such as

-   printing the generated configurations as command line options,

-   running a batch of experiments based on the generated configurations,

-   store the results of the experiments in a database,

-   running a parameter race (see ) to find out the configurations (or configuration) that optimize a function of quality,

-   retrieving the results of the experiments from the database.

In the first case, the outcome of our above example would be something like this (**json2run** comes in form of a command line tool called `j2r`):

    $ j2r -i experiments.json
    --a foo --b1 0.0
    --a foo --b1 0.25
    ...
    --a baz --b2 8.0
    --a baz --b2 10.0

**json2run** supports a number of different types of nodes in the JSON tree, including: nodes for generating parameters from a discrete set of variables, nodes for generating parameters from the content of a directory or a file, nodes for sampling values from an interval, and so on. If something cannot be expressed with simple parameter generators, a number of **post-processors** allow you to mix, merge and discard the generated parameters in extremely flexible ways. Most post-processors were created because something couldn't be expressed with simple logical operators, but **json2run** slowly converged to something complete now.

The experiments results are stored on a (MongoDB) database, in order to be accessed later for analysis. The choice of MongoDB comes from the necessity of comparing algorithms which can have different (and a different number of) parameters, and a using a tabular storage (as in most relational database) would make queries much more difficult.

Finally, **json2run** comes with a very general but handy R script, which allows to gather data from the database, and do whatever kind of statistical analysis over it.

# Installation

Being packaged as a python module, the installation of **json2run** should be quite straightforward, just ensure that the `bson` python module is not installed on your system (if this is the case, run `sudo pip uninstall bson`) since `pymongo` comes with its own `bson.*` classes and conflicts may occur. Then, clone the (Mercurial) repository and run the python installer

or, if you plan to update **json2run** often (i.e. if you plan to customize it, or update it from the repository), run

    sudo python setup.py develop

this will allow you to update to the latest version by just running

    hg pull -u

in the root directory.

# Usage

Since **json2run** is designed to be very flexible (its only requirements being that you expose all the parameters of your executable and that you have access to a MongoDB instance), this also means that it comes with a lot of options. We will go through them in the following sections, but if you just need a quick reference type

    $ j2r --help

Before running anything, however, you will need to know how to write an experiment file.

## Designing experiment files

As previously mentioned, **json2run** expects a description of the experiments in JSON format. JSON (which, by the way, stands for JavaScript Object Notation) is a concise and human-readable language for describing structured data. It is also the very language MongoDB uses to store its data and to make queries, which makes for a natural integration.

### Basic JSON syntax

The basic components of a JSON documents are arrays, objects and scalars. You can use array and object to group more arrays and objects, or scalars.

#### Scalars

JSON has a number of native types for scalars:

-   numbers,

-   strings, and

-   booleans.

Numbers can be integers or floats, and scientific notation is also supported.

#### Arrays and objects

Arrays are lists of scalars separated by commas, e.g.:

    [1, 2, 3, "foo", 3.14, true]

Objects can be seen as named arrays (similar to C++'s map, or dictionaries if you're into Python):

    {
        "name": "John",
        "surname": "Boags",
        "profession": "beer maker",
        "age": 150
    }

Note that arrays and objects use a different kind of parenthesis, `{}` vs `[``]`.

#### Comments

JSON doesn't support comments, and most of the time you won't need them (the JSON contents should be sufficiently explanatory), but if you really need annotations, you can add fake entries to your objects, that won't be parsed by **json2run** and will serve as comments, such as `comment` in the following example:

#### Specific syntax

In particular, **json2run** assumes that the experiment file is a representation of a tree in which each node is a JSON object with at least a `type` field describing its type, e.g.:

    {
        "type": "<node_type>",
        ...
    }

We'll see the supported node types and their additional fields in the following sections.

### Combinations and alternatives (inner nodes)

Typically, an algorithm accepts multiple parameters, and we want to be able to compare alternative combinations of these parameters. `and` and `or` nodes are the way to accomplish this in **json2run** and we call them *inner* nodes. Each inner node has a list of descendants and when they are activated, they either combine them together (in case of an `and`), or pick between them (in case of an `or`).

#### `and` nodes

For instance, suppose that we have a Simulated Annealing solver that accepts an initial temperature and a cooling schedule as command line parameters, and we would like to try all the possible combinations. In **json2run** this is expressed using an `and` node that combines the two parameters (ignore for now the syntax to describe discrete parameters, we'll come to that later).

#### `or` nodes

Sometimes you have algorithms that accept different parameters and, possibly, a different number of them. Suppose your solver can operate either using Simulated Annealing and Tabu Search. You can easily encode this in an experiment file by using an `or` node.

Note that no Tabu Search parameters appear in the Simulated Annealing configurations, and vice versa. By combining several `and` and `or` node, it is possible to express quite complex experiment designs.

### Leaf nodes

Leaf nodes are responsible for creating values for single parameters. They all come with a `type`, a `name` for the parameter, and an array (or object) of `values` describing the possible values that the parameter can take, e.g.:

    {
        "type": "<leaf_type>",
        "name": "<parameter_name>",
        "values": <values_definition>
    }

#### `discrete` nodes

Discrete nodes are the simplest kind of leaf nodes. They come with two value definition styles: an *explicit* one, where parameters are listed explicitly, e.g.:

    {
        "type": "discrete",
        "name": "num_of_reheats",
        "values": [ 1, 2 ]
    }

and an *implicit* one, which allows to define discrete numeric values from a `min`, a `max` and a `step`, e.g.:

During the processing of the tree, implicit value definitions are transformed into explicit ones, and treated as such. A `discrete` node generates all the possible parameter values in order.

#### `continuous` nodes

Continuous nodes allow to define parameter values that are generated by sampling continuous parameter spaces. These spaces are defined in terms of a `min` and a `max`, e.g.:

    {
        "type": "continuous",
        "name": "start_temperature",
        "values": { "min": 0.0, "max": 10.0 }
    }

However, continuous nodes can't generate parameter values by themselves. Instead, they need to be processed later on by a *post-processor* attached to a node upper in the tree hierarchy. This might seem over complicated, but there's a use case behind it.

In particular suppose that you want to study the interaction of two parameters on the performance of an algorithm. To do a proper sampling, the generated parameters must be picked from a 2-dimensional space, in a way that is as uniform a possible. One way to do this, would be to generate an `and` node containing several `discrete` nodes with different ranges. There are two problems with this approach (which is called *full-factorial*):

1.  the generated points are very regular, while one usually want to sample the parameter space randomly (but uniformly),

2.  the parameter combinations are the carthesian product of the values generated for each single parameter, which makes it difficult to control how much configurations are generated.

While this can still be done with **json2run** (and indeed is often what one wants), we would like to treat the parameter space generated by the interaction of the two parameters as a *single* space, and sample 2-dimensional points uniformly inside it. **json2run** comes with a post-processor which is able to generate the Hammersley point set in a k-dimensional space. We will see in the section about post-processors how to attach one to a node, but for the moment just accept that `continuous` leaf nodes are treated in this special way.

#### `file` and `directory` nodes

File (or directory) nodes are essentially `discrete` nodes, whose values are nor defined explicitly nor implicitly, but instead are generated from the content of a file (or a directory). The typical use for this kind of nodes is to generate experiments that run on a set of instances specified in a file e.g.:

    {
        "type": "file",
        "name": "instance",
        "path": "selected_instances.txt",
        "match": ".*"
    }

where the `path` field specifies the location of the file to be used as input, and the `match` field restrict the generated parameters to the lines of the file matching a given regular expression[^2]. This is useful when you want to restrict to certain instances, but most frequently will just be ".\*" (catch-all). As for directory nodes, they follow a similar semantic, the difference being that the generated values is the list of the content of the directory, filtered by the regular expression in the `match` field.

    {
        "type":"directory",
        "path": "../instances/comp",
        "name": "instance",
        "match": ".*\\.ectt"
    }

#### `flag` nodes

Flag nodes have a single parameter (the `name` of the generated flag) and generate value-less parameters. E.g.:

    {
        "type": "flag",
        "name": "verbose"
    }

will just add the `--verbose` flag on the generated command lines.

### Post-processors

Post-processors are tools to generate more complex combinations of parameters. They can be attached to **any** inner node by adding them to a field called `postprocessors` along with the descendants, e.g.:

    {
        "type": "and",
        "descendants": [
            ...
        ],
        "postprocessors": [
            ...
        ]
    }

Post-processors have a `type` field and a number of other fields dependent on the specific post-processor type. They all operate in the following way:

1.  They take the list of parameters generated in the subtree (note that each execution of a subtree gives birth to a different parameter configuration) they are attached to,

2.  they process the list **as a whole** (e.g. replacing parameters, modifying values, removing parameters, and so forth), and finally

3.  they return the new list, which replaces the old one.

In many cases they just apply the same function over all the elements of the list, but they might be designed to do more complex things or to just update certain kind of parameters. For instance, the `hammersley` post-processor only apply to `continuous` parameters (but possibly more than one of them at a time).

**Note** order matters! Post-processors are applied in the order in which they are defined in the `postprocessors` list.

#### `expression` processors

Expression post processors are by far the most flexible ones. They allow to define a new parameter (either `discrete` or `continuous`, but **not** a flag) by evaluating a python expression and using the result as the value of the parameter. The processor is defined by a `match` regular expression, which captures the operands needed by the expression, and either

1.  an `expression` which will be evaluated to yield the value of the generated `discrete` parameter, or

2.  two expressions (`min` and `max`) that will yield the values for the minimum and maximum of the generated `continuous` parameter.

The type of the parameter is inferred by the presence of the `expression` field, while its name is defined by the `result` field. An example of the two syntaxes is the following:

    {
        "type": "expression",
        "match": "<operand_1>|<operand_2>|..."
        "result": "<parameter_name>",
        "expression": "<expression>"
    }

    {
        "type": "expression",
        "match": "<operand_1>|<operand_2>|...",
        "result": "<parameter_name>",
        "min": "<min_expression>",
        "max": "<max_expression>"
    }

#### Expression syntax

Any valid python expression that has a return value can be used as `expression`, `min` or `max`. To access the values of the captured operands it is sufficient to postfix their name with `.value`.

As for the available operations, functions from python's `math` and `json` modules are automatically imported. For instance, to generate a new parameter *p3* which is the power of two existing ones, *p1* and *p2*, we'll write:

    {
        "type": "expression",
        "match": "p1|p2",
        "result": "p3",
        "expression": "pow(p1.value, p2.value)"
    }

While to generate a parameter which takes values in *\[0.1\*p1, 5\*p2\]*, we'll do:

    {
        "type": "expression",
        "match": "p1|p2",
        "result": "p3",
        "min": "0.1*p1.value",
        "max": "5*p2.value"
    }

#### `ignore` processors

Ignore post processors can be used to remove specific parameters from the list of generated ones. Typically, the are used to discard operands of an `expression` post-processor after they have been used. Following the previous example, we might not be interested in *p1* and *p2* at all, so:

    {
        "type": "ignore",
        "match": "p1|p2"
    }

Remember that post-processors are applied in order, thus (in this case) the `ignore` must be defined after the `expression`.

#### `sorting` processors

Sorting allows to define an ordering for a subset of parameters. These parameters will be put (if they exist) at the beginning of the generated list of parameters, and the others will follow. The syntax of the post-processor is the following:

    {
        "type": "sorting",
        "order": [ "<param_1>", "<param_2>", ... ]
    }

Where `order` is an array of ordered parameter names.

#### `hammersley` processors

The Hammersley post-processor generates the scaled k-dimensional Hammersley point set from a set of k `continuous` parameters, and it's the preferential (also, the only) way to sample continuous parameter spaces. The syntax is the following:

    {
        "type": "hammersley",
        "points": <n>
    }

So, assuming that your experiments file produces *k* continuous parameters, the `hammersley` post-processor generates a k-dimensional *cube* delimited by the `min` and `max` fields of your `continuous` parameters, and will generate `n` samples inside this cube, to use as parameter values.

#### The Hammersley point set

This choice of the Hammersley point set has been driven by two properties of this sequence that make it favourable to parameter tuning. First, points from the Hammersley set exhibit *low discrepancy*, i.e. they are well distributed across the parameter space despite being random-like. Second, the sequence is *scalable* both with respect to the number of points (`n`) to be sampled and to the number of dimensions (`k`) of the sampling space.

So, whenever you want to explore parameter spaces, use `continuous` parameters and the `hammersley` post-processor.

#### `rounding` processors

When sampling continuous parameters or using `expression` post-processors, the resulting values can end up being floats with many decimal digits. While this in general is not an issue, often this much precision is unneeded, and it's just more convenient to operate with less precise floats. The `rounding` post-processor allows to round down a parameter's values to a specific number of decimal digits. The syntax is the following:

    {
        "type": "rounding",
        "match": "<regex>"
        "decimal_digits": <n>
    }

Where `n` is the number of decimal digits we want to retain (**note** the numbers are rounded, not truncated, to `n` digits after the floating point), and `match` is a regular expression describing the parameters we want to round down.

#### Compact syntax

Rounding post-processors also support a compact syntax, to group roundings in a single post-processor. The syntax is the following:

    {
        "type": "round",
        "round": [
            "<regex_1>": <n_1>,
            "<regex_2>": <n_2>,
            ...
        ]
    }

Where `regex_k` are regular expressions describing one (or more) parameter, and `n_k` are the corresponding decimal digits we want to retain.

#### Forcing precision

Both syntaxes support an optional field called `force_precision`, which can be either `true` or `false`, that forces the resulting value to have the specified number of decimal digits (regardless of any rounding to zero).

#### `renaming` processors

Rename post-processors can be used to rename parameters. The syntax, somewhat similar to `rounding`'s compact one, is the following:

    {
        "type": "renaming",
        "rename": {
            "old_1": "new_1",
            "old_2": "new_2",
            ...
        }
    }

Where `old_`$k$ are the original name of the parameters we want to rename and `new_k` are the new ones. Note that unlike `rounding`'s compact syntax, here `rename` is an object, not an array. Also, `old_k` are plain strings, not regular expressions.

## The `j2r` command line tool

All of **json2run** functionalities are accessed through a command line utility called `j2r`. The tool comes with a (large) number of options, activated by `--` followed by their long names, or equivalently `-` followed by their short names. Some of them have default values, some other must be provided. (See `j2r --help` for a summary of them.)

### Input

Most of the functionalities of **json2run** require that you provide an input file, i.e. a JSON file describing experiments. This file is specified through the `--input` (or `-i`) option.

    $ j2r -i experiments.json

### Available actions

The main switch in `j2r` is the `--action` (or `-a`) option. Actions allow to specify what you want **json2run** to do for you. The available actions are:

-   `print-cll` prints the generated experiments as a *command line list* (also, the default option)

-   `print-csv` print the generated experiments as a CSV file

-   `run-batch` start (or resumes) a batch for the experiments described in the input file

-   `run-race` starts (or resumes) a race among parameter configurations in order to find the best parameter assignment for the specified executable

-   `list-batches` list the batches on the database, and give summary information about their completion, machine on which they are being run, type of batch, etc.

-   `delete-batch` delete a batch from the database

-   `batch-info` provides detailed information about a batch or race, e.g. repetitions completed, configurations that are still racing, experiment file, etc.

-   `rename-batch` rename a batch on the database

-   `show-winning` show winning configurations in a race

-   `set-repetitions` set the number of repetitions of the same experiments in a batch or a race

-   `dump-experiments` dump all the experiments data regarding a batch as a CSV file

-   `mark-unfinished` set a batch as unfinished, in order to restart it

### Batch options

Some of the actions require additional parameters, in particular, each action pertaining a batch on the database must also provide a mandatory `--batch-name` (or `-n`) to refer it (see it as a key for batches in the database).

### Running options

Both `run-batch` and `run-race` have a number of additional parameters that tune the way in which the experiments are run.

-   `--executable` (or `-e`) specifies the executable to be run (mandatory)

-   `--parallel-threads` (or `-p`) specifies the maximum number of parallel processors to run the experiments onto (defaults to the number of cores on the machine where the experiments are run)

-   `--repetitions` (or `-r`) number of repetitions of the (exactly) same experiment to run (e.g. to have a more reliable result)

-   `--greedy` (or `-g`) can be `true` or `false` and states if the batch or race can reuse experiments which are already on the database (but are possibly part of other batches and races)

### Extra options for races

When a race is run, a number of additional parameters must or can be passed.

-   `--instance-param` (or `-ip`) specifies the parameter which represents the instance

-   `--performance-param` (or `-pp`) specifies the statistic (output by the executable), that must be used to evaluate the quality of a configuration

-   `--seed` (or `-s`) seed to use to shuffle instances (defaults to 0)

-   `--confidence` confidence level for hypothesis testing (e.g. to compare with p-values, defaults to *0.05*)

**Note** **json2run** assumes that the executable outputs a valid JSON code, with a field for each statistic that we want to record. For instance, a solver could have a `cost` and a `time` statistic.

    {
        "cost": 161.12,
        "time": 500
    }

### Database options

When we're running batches or databases, we're implicitly assuming that we have a running and accessible MongoDB database. By default, **json2run** will look for MongoDB on the `localhost` and will try to connect to the database `j2r` with username `j2r` and password `j2r`. These are just convenient credentials, but one can specify its own connection parameters through the following options.

-   `--db-host` (or `-dh`) specifies the host onto which the MongoDB instance is running

-   `--db-database` (or `-dd`) specifies the database to use connecting

-   `--db-user` (or `-du`) specifies the username to use for connecting

-   `--db-pass` (or `-dx`) specifies the password to use for connecting

-   `--db-port` (or `-dp`) specifies the port to use for connecting

### Logging info

By default `j2r` prints on the standard output most of its logging information. However this information can be redirected on a file if needed, and the log level can be set.

-   `--log-file` specifies the file where the log is written (default: None)

-   `--log-level` can be `warning`, `error`, 'info

### Source code versioning

Additionally, **json2run** can record the code revision used for running a batch or a race. To enable this option one must pass the name of the source code manager of choice through the `--scm` option (currently supports `git` and `mercurial`).

### Instances and configurations

Instances and parameter configurations are described in the same experiments file.

## Running examples

Here are some of the most common operations that one can perform with **json2run**.

### Running a batch of experiments

Run a batch of experiments based on an experiment file (*experiments.json*) and an executable (*solver*), with 10 repetitions for each experiment and all the available cores.

### Running a configuration race

Based on the same file, and reckoning that the instance parameter is called *instance*, we can run a race to find out the best configuration. Suppose that the solver outputs some statistics in JSON (as in the example above) and that we want to compare the configurations based on the *cost* of the obtained solutions.

### Resuming a batch or a race

To resume a previously stopped batch or race, it is sufficient to run

    $ j2r -a run-batch -n my_batch

or

    $ j2r -a run-race -n my_race

### Printing detailed data about a batch or race

Use the `batch-info` action, passing the name of the race or batch.

    $ j2r -a batch-info -n my_race

the output is in JSON format (for easy parsing by other tools).

### Print the list of winning (so far) configurations in a race

Use the `show-winning` action, passing the name of the race or batch.

    $ j2r -a show-winning -n my_race

### Delete a batch or a race from the database

Use the `delete-batch` action, passing the name of the race or batch.

    $ j2r -a delete-batch -n my_race

### List all the batches on the database

Use the `list-batches` action.

    $ j2r -a list-batches

## Analyzing the outcome

The outcome of a batch or race, i.e. all the data about the experiments, can be retrieved from R by loading the R script `analysis.R` and using the following functions:

    source("analysis.R")
    connect("host")

    x <- getExperiments("my_race", c("instance"))

The `x` data frame will contain a row for each experiment in the batch or race, with information about whether the configuration was one of the winning ones (in case of a race).

# Future

A new major version of **json2run** is in the works. Among the upcoming features are:

-   launching of experiments on multiple machines,

-   web-service, i.e. RESTful, infrastructure will handle all **json2run** operations,

-   improved JSON syntax for all node types (fields and type of values will determine which kind of node are we dealing with), e.g.:

    {
        "type": "and",
        "descendants": [ ... ]
    }

will become:

    {
        "and": [ ... ]
    }

and

will become:

# Licensing

**json2run** is open-source and distributed under the MIT license.

# Acknowledgements

**json2run** has been developed for and with the collaboration of Luca Di Gaspero, Sara Ceschia and Andrea Schaerf of the Scheduling and Time-Tabling Group of University of Udine. Thanks go to Tiago Januario from Universidade Federal de Minas Gerais, Belo Horizonte, Brasil for the many suggestions. Also, thanks go to the [Bitbucket](http://www.bitbucket.org) staff that hosts **json2run**'s code free of charge.

[^1]: *Scheduling and Time-Tabling Group*, DIEGM - University of Udine, Via delle Scienze 206, 33100 – Udine (UD), Italy

[^2]: **Note** regular expressions are in Python format, but strings must be escaped, e.g. if you want to look for the .txt pattern, the string must be specified as ".\*\\.txt"