title: LEVELS OF REALITY AS SOURCE OF QUANTUM INDETERMINACY[^1]

Basarab NICOLESCU

Laboratoire de Physique Nucléaire et de Hautes Energies (LPNHE) [^2]
LPTPE, Tour 12 E3 - 4, Place Jussieu 75252 Paris Cedex 05, France
e-mail: email@example.com

# Quantum physics and levels of Reality

The major cultural impact of the quantum physics has certainly raised questions for the contemporary philosophical dogma of the existence of a single level of Reality .

Here the meaning we give to the word "Reality" is pragmatic and ontological at the same time.

By Reality I intend first of all to designate that which resists our experiences, representations, descriptions, images or mathematical formalizations. Quantum physics caused us to discover that abstraction is not simply an intermediary between us and Nature, a tool for describing reality, but rather, one of the constituent parts of Nature. In quantum physics, mathematical formalization is inseparable from experience. It resists in its own way by its simultaneous concern for internal consistency, and the need to integrate experimental data without destroying that self-consistency.

In so far as Nature participates in the being of the world one must ascribe an ontological dimension to the concept of Reality. Nature is an immense, inexhaustible source of the unknown which justifies the very existence of science. Reality is not only a social construction, the consensus of a collectivity, or an intersubjective agreement. It also has a trans-subjective dimension, to the extent that one simple experimental fact can ruin the most beautiful scientific theory.

By *level of Reality* I intend to designate an ensemble of systems which are invariant under the action of certain general laws: for example, quantum entities are subordinate to quantum laws, which depart radically from the laws of the macrophysical world. That is to say that two levels of Reality are *different* if, while passing from one to the other, there is a break in the laws and a break in fundamental concepts (like, for example, causality). No one has succeeded in finding a mathematical formalism which permits the rigorous passage from one world to another. Semantic glosses, tautological definitions or approximations are unable to replace a rigorous mathematical formalism. The recent decoherence models have nothing precise to say on the passage between the quantum level and the macrophysical level: in fact, the main problem is not decoherence but precisely *coherence*.

There are even strong mathematical indications that the continuous passage from the quantum world to the macrophysical world would never be possible. But there is nothing catastrophic about this. The *discontinuity* which is manifest in the quantum world is also manifest in the structure of the levels of Reality. That does not prevent the two worlds from co-existing.

The levels of Reality are radically different from the levels of organization as these have been defined in systemic approaches . Levels of organization do not presuppose a break with fundamental concepts: several levels of organization appear at one and the same level of Reality. The levels of organization correspond to different structurings of the same fundamental laws. For example, Marxist economy and classical physics belong to one and the same level of Reality.

The emergence of at least two different levels of Reality in the study of natural systems is a major event in the history of knowledge.

The existence of different levels of Reality has been affirmed by different traditions and civilizations, but these affirmations were founded on religious dogma or on the exploration of the interior universe.

In our century, in their questioning of the foundations of science, Edmund Husserl and other scholars have discovered the existence of different levels of perception of Reality by the subject-observer. But these thinkers, pioneers in the exploration of a multi-dimensional and multi-referential reality, have been marginalized by academic philosophers and misunderstood by the majority of physicists, enclosed in their respective specializations. The view I am expressing here is totally conform to the one of Heisenberg, Pauli and Bohr.

In fact, Werner Heisenberg came very near, in his philosophical writings, to the concept of "level of Reality". In his famous *Manuscript of the year 1942* (published only in 1984) Heisenberg, who knew well Husserl, introduces the idea of three *regions of reality*, able to give access to the concept of "reality" itself: the first region is that of classical physics, the second - of quantum physics, biology and psychic phenomena and the third - that of the religious, philosophical and artistic experiences . This classification has a subtle ground: the closer and closer connectiveness between the Subject and the Object.

As we shall see in the following, the notion of levels of Reality will lead us to a general philosophical understanding of the nature of indeterminacy. If there was only one region or level of reality, it was impossible to conceive what means a true, irreducible indeterminacy, like the quantum one.

# The logic of the included middle

Knowledge of the coexistence of the quantum world and the macrophysical world and the development of quantum physics has led, on the level of theory and scientific experiment, to the upheaval of what were formerly considered to be *pairs of mutually exclusive contradictories* (A and non-A): wave *and* corpuscle, continuity *and* discontinuity, separability *and* nonseparability, local causality *and* global causality, symmetry *and* breaking of symmetry, reversibility *and* irreversibility of time, etc.

The intellectual scandal provoked by quantum mechanics consists in the fact that the pairs of contradictories that it generates are actually mutually contradictory when they are analyzed through the interpretative filter of classical logic. This logic is founded on three axioms:

1.  *The axiom of identity* : A is A.

2.  *The axiom of non-contradiction* : A is not non-A.

3.  *The axiom of the excluded middle* : There exists no third term T which is at the same time A and non-A.

Under the assumption of the existence of a single level of Reality, the second and third axioms are obviously equivalent.

If one accepts the classical logic one immediately arrives at the conclusion that the pairs of contradictories advanced by quantum physics are mutually exclusive, because one cannot affirm the validity of a thing and its opposite at the same time: A *and* non-A.

Since the definitive formulation of quantum mechanics around 1930 the founders of the new science have been acutely aware of the problem of formulating a new, "quantum logic." Subsequent to the work of Birkhoff and van Neumann a veritable flourishing of quantum logics was not long in coming . The aim of these new logics was to resolve the paradoxes which quantum mechanics had created and to attempt, to the extent possible, to arrive at a predictive power stronger than that afforded by classical logic.

Most quantum logics have modified the second axiom of classical logic – the axiom of non-contradiction – by introducing non-contradiction with several truth values in place of the binary pair (A, non-A). These multivalent logics, whose status with respect to their predictive power remains controversial, have not taken into account one other possibility: the modification of the third axiom – the axiom of the excluded middle.

History will credit Stéphane Lupasco with having shown that the *logic of the included middle* is a true logic, formalizable and formalized, multivalent (with three values: A, non-A, and T) and non-contradictory . His philosophy, which takes quantum physics as its point of departure, has been marginalized by physicists and philosophers. Curiously, on the other hand, it has had a powerful albeit underground influence among psychologists, sociologists, artists, and historians of religions. Perhaps the absence of the notion of "levels of Reality" in his philosophy obscured its substance: many persons wrongly believed that Lupasco's logic violated the principle of non-contradiction.

Our understanding of the axiom of the included middle – *there exists a third term T which is at the same time A and non-A* – is completely clarified once the notion of "levels of Reality" is introduced.

In order to obtain a clear image of the meaning of the included middle, we can represent the three terms of the new logic – A, non-A, and T – and the dynamics associated with them by a triangle in which one of the vertices is situated at one level of Reality and the two other vertices at another level of Reality. If one remains at a single level of Reality, all manifestation appears as a struggle between two contradictory elements (example: wave A and corpuscle non-A). The third dynamic, that of the T-state, is exercised at another level of Reality, where that which appears to be disunited (wave or corpuscle) is in fact united (quanton), and that which appears contradictory is perceived as non-contradictory.

It is the projection of T on one and the same level of Reality which produces the appearance of mutually exclusive, antagonistic pairs (A and non-A). A single level of Reality can only create antagonistic oppositions. It is inherently *self-destructive* if it is completely separated from all the other levels of Reality. A third term, let us call it $T_{0}$, which is situated on the same level of Reality as that of the opposites A and non-A, can not accomplish their reconciliation.

The T-term is the key in understanding indeterminacy: being situated on a different level of Reality than A and non-A, it necessarily induces an *influence* of its own level of Reality upon its neighbouring and different level of Reality: *the laws of a given level are not self-sufficient to describe the phenomena occuring at the respective level*.

The entire difference between a triad of the included middle and an Hegelian triad is clarified by consideration of the role of *time*. *In a triad of the included middle the three terms coexist at the same moment in time*. On the contrary, each of the three terms of the Hegelian triad succeeds the former in time. This is why the Hegelian triad is incapable of accomplishing the reconciliation of opposites, whereas the triad of the included middle is capable of it. In the logic of the included middle the opposites are rather *contradictories* : the tension between contradictories builds a unity which includes and goes beyond the sum of the two terms. The Hegelian triad would never explain the nature of indeterminacy.

One also sees the great dangers of misunderstanding engendered by the common enough confusion made between the axiom of the excluded middle and the axiom of non-contradiction . The logic of the included middle is non-contradictory in the sense that the axiom of non-contradiction is thoroughly respected, a condition which enlarges the notions of "true" and "false" in such a way that the rules of logical implication no longer concerning two terms (A and non-A) but three terms (A, non-A and T), co-existing at the same moment in time. This is a formal logic, just as any other formal logic: its rules are derived by means of a relatively simple mathematical formalism.

One can see why the logic of the included middle is not simply a metaphor, like some kind of arbitrary ornament for classical logic, which would permit adventurous incursions into the domain of complexity. *The logic of the included middle is the privileged logic of complexity*, privileged in the sense that it allows us to cross the different areas of knowledge in a coherent way, by enabling a new kind of simplicity.

The logic of the included middle does not abolish the logic of the excluded middle: it only constrains its sphere of validity. The logic of the excluded middle is certainly valid for relatively simple situations. On the contrary, the logic of the excluded middle is harmful in complex, transdisciplinary cases. For me, the problem of indeterminacy is precisely belonging to this class of cases.

# The Gödelian unity of the world

The transdisciplinary approach sets forth for consideration a multi-dimensional Reality, structured by multiple levels replacing the single level of classical thought – one-dimensional reality. This proposal is not enough, by itself, to justify a new vision of the world. We must first of all answer many questions in the most rigorous possible way. What is the nature of the theory which can describe the passage from one level of Reality to another? Is there truly a coherence, a unity of the totality of levels of Reality? What is the role of the subject-observer of Reality in the dynamics of the possible unity of all the levels of Reality? Is there a level of Reality which is privileged in relation to all other levels? What is the role of reason in the dynamics of the possible unity of knowledge? What is the predictive power of the new model of Reality in the sphere of reflection and action? Finally, is understanding of the present world possible ?

According to our model, Reality comprises a certain number of levels \[1,2\]. The considerations which follow do not depend on whether or not this number is finite or infinite. For the sake of clarity, let us suppose that this number is infinite.

Two adjacent levels are connected by the logic of the included middle in the sense that the T-state present at a certain level is connected to a pair of contradictories (A and non-A) at the immediately adjacent level. The T-state operates the unification of contradictories A and non-A but this unification is operated at a level *different* from the one on which A and non-A are situated. The axiom of non-contradiction is thereby respected. Does this fact signify that we can obtain a complete theory, which will be able to account for all known and forthcoming results ?

There is certainly a coherence between different levels of Reality, at least in the natural world. In fact, an immense *self-consistency* – a cosmic bootstrap – seems to govern the evolution of the universe, from the infinitely small to the infinitely large, from the infinitely brief to the infinitely long . A flow of information is transmitted in a coherent manner from one level of Reality to another level of Reality in our physical universe.

The logic of the included middle is capable of describing the coherence between the levels of Reality by an iterative process defined by the following stages: 1. A pair of contradictories (A, non-A) situated at a certain level of reality is unified by a T-state situated at a contiguous level of Reality; 2. In turn, this T-state is linked to a couple of contradictories (A', non-A'), situated at its own level; 3. The pair of contradictories (A', non-A') is, in its turn, unified by a T'-state situated at a different level of Reality, immediately contiguous to that where the ternary (A', non-A', T) is found. The iterative process continues indefinitely until all the levels of Reality, known or conceivable, are exhausted.

In other terms, the action of the logic of the included middle on the different levels of Reality induces an *open*, *Gödelian* structure of the unity of levels of Reality. This structure has considerable consequences for the theory of knowledge because it implies the impossibility of a complete theory, closed in upon itself.

In effect, in accordance with the axiom of non-contradiction, the T-state realizes the unification of a pair of contradictories (A, non-A) but it is associated, at the same time with another pair of contradictories (A', non-A'). This signifies that starting from a certain number of mutually exclusive pairs one can construct a new theory which eliminates contradictions at a certain level of Reality, but this theory is only temporary because it inevitably leads, under the joint pressure of theory and experience, to the discovery of new levels of contradictories, situated at a new level of Reality. In turn this theory will therefore be replaced by still more unified theories as new levels of Reality are discovered. This process will continue indefinitely without ever resulting in a completely unified theory. The axiom of non-contradiction is increasingly strengthened during this process. In this sense, without ever leading to an absolute non-contradiction, we can speak of an *evolution of knowledge* which encompasses all the levels of Reality: knowledge which is forever open. Finer matter penetrates coarser matter, just as quantum matter penetrates macrophysical matter, but the reverse is not true. *Degrees of materiality* induce an orienting arrow for tracing the transmission of information from one level to the other. This orienting arrow is associated with the discovery of more and more general, unifying, and encompassing laws.

The open structure of the unity of levels of Reality is in accord with one of the most important scientific results of the 20th century concerning arithmetic, the theorem of Kurt Gödel . Gödel's theorem tells us that a sufficiently rich system of axioms inevitably lead to results which would be either undecidable or contradictory. The implications of Gödel's theorem have considerable importance for all modern theories of knowledge. First of all it does not only concern the field of arithmetic but also all mathematics which includes arithmetic. Now, obviously the mathematics which underlies theoretical physics include arithmetic. This means that all research for a complete physical theory is illusory.

In fact, the search for an axiomatic system leading to a complete theory (without undecidable or contradictory results) marks at once the apex and the starting point of the decline of classical thought. The axiomatic dream is unraveled by the verdict of the holy of holies of classical thought – mathematical rigor.

The theorem that Kurt Gödel demonstrated in 1931 sounded only a faint echo beyond a very limited circle of specialists. The difficulty and extreme subtlety of its demonstration explains why this theorem has taken a certain time to be understood within the mathematical community. Today, it has scarcely begun to penetrate the world of physicists. Wolfgang Pauli, one of the founders of quantum mechanics, was one of the first physicists to understand the extreme importance Gödel's theorem has for the construction of physical theories .

The Gödelian structure of the unity of levels of Reality associated with the logic of the included middle implies that it is impossible to construct a complete theory for describing the passage from one level to the other and, *a fortiori*, for describing the unity of levels of Reality.

If it does exist, the unity linking all the levels of Reality must necessarily be an open *unity*.

To be sure, there is a coherence of the unity of levels of Reality, but we must remember that this coherence is *oriented* : there is an arrow associated with all transmission of information from one level to the other. As a consequence of this, if coherence is limited only to the levels of Reality, it is stopped at the "highest" level and at the "lowest" level. If we wish to posit the idea of a coherence which continues beyond these two limited levels so that there is an open unity, one must conceive the unity of levels of Reality as a unity which is extended by a *zone of non-resistance* to our experiences, representations, descriptions, images and mathematical formalizations. Within our model of Reality, this zone of non-resistance corresponds to the "veil" which Bernard d'Espagnat referred to as "the veil of the real" . The "highest" level and the "lowest" level of the unity of levels of Reality are united across a zone of absolute transparence. But these two levels are different; from the point of view of our experiences, representations, descriptions, images, and mathematical formalizations, absolute transparence functions like a veil. In fact, the open unity of the world implies that that which is "below" is the same as that which is "above". The isomorphism between "above" and "below" is established by the zone of non-resistance.

Quite simply, the non-resistance of this zone of absolute transparence is due to the limitations of our bodies and of our sense organs, limitations which apply regardless of the instruments of measure used to extend these sense organs. To claim that there is an infinite human knowledge (which excludes any zone of non-resistance), while simultaneously affirming the limitations of our body and our sense organs, seems to us a feat of linguistic sleight of hand. The zone of non-resistance corresponds to the sacred, *that is to say to that which does not submit to any rationalization*.

The unity of levels of Reality and its complementary zone of non-resistance constitutes *the transdisciplinary Object*.

A new *Principle of Relativity* emerges from the coexistence between complex plurality and open unity : *no one level of Reality constitutes a privileged place from which one is able to understand all the other levels of Reality*. A level of Reality is what it is because all the other levels exist at the same time. This Principle of Relativity is what originates a new perspective on religion, politics, art, education, and social life. In the transdisciplinary vision, Reality is not only multi-dimensional, it is also multi-referential.

The different levels of Reality are accessible to human knowledge thanks to the existence of different *levels of perception*, which are in bi-univocal correspondence with levels of Reality. These levels of perception permit an increasingly general, unifying, encompassing vision of Reality, without ever entirely exhausting it.

As in the case of levels of Reality the coherence of levels of perception presupposes a zone of non-resistance to perception.

The unity of levels of perception and its complementary zone of non-resistance constitutes *the transdisciplinary Subject*.

The two zones of non-resistance of transdisciplinary Object and Subject must be *identical* in order that the transdisciplinary Subject can communicate with the transdisciplinary Object. *A flow of consciousness crossing the different levels of perception in a coherent manner must correspond to the flow of information crossing the different levels of Reality in a coherent manner*. The two flows are in a relation of *isomorphism* thanks to the existence of one and the same zone of non-resistance. Knowledge is neither exterior nor interior: it is *at the same time* exterior and interior. The study of the universe and the study of the human being sustain one another. The zone of non-resistance permits the unification of the transdisciplinary Subject and the transdisciplinary Object while preserving their difference.

Transdisciplinarity is the transgression of duality opposing binary pairs: subject/object,subjectivity/objectivity, matter/consciousness,
nature/divine, simplicity/complexity, reductionism/holism, diversity/unity. This duality is transgressed by the open unity which encompasses both the universe and the human being.

The transdisciplinary model of Reality has, in particular, some important consequences in the study of *complexity*. Without its contradictory pole of simplicity (or, more precisely, *simplexity*) complexity appears as an increasingly enlarging *distance* between the human being and Reality which introduces a self-destructive alienation of the human being who is plunged into the absurdity of destiny. The infinite simplicity of the transdisciplinary Subject corresponds to the infinite complexity of the transdisciplinary Object.

The Subject/Object problem was central for the founding-fathers of quantum mechanics. Pauli, Heisenberg and Pauli, as Husserl, Heidegger and Cassirer refuted the basic axiom of modern metaphysics: the clear-cut distinction between Subject and Object. Our considerations here are inscribed in the same framework.

# The death and the resurrection of Nature

Modernity is particularly deadly. It has invented all kinds of "deaths" and "ends": the death of God, the death of Man, the end of ideologies, the end of history and, today, the end of science .

But, there is a death which is spoken of much less, on account of shame or ignorance : *the death of Nature*. In my view, this death of Nature is the source of all the other deadly concepts which were just invoked. In any case, the very word "Nature" has ended by disappearing from scientific vocabulary. Of course, the "man in the street", just as the scientist (in popularized works) still uses this word, but in a confused, sentimental way, reminiscent of magic.

Since the beginning of time we have not stopped modifying our vision of Nature . Historians of science are in accord in saying that, despite all appearances to the contrary, there is not only one vision of Nature across time. What can there be in common between the Nature of so-called "primitive" peoples, the Nature of the Greeks, the Nature in the time of Galileo, of the Marquis de Sade, of Laplace or of Novalis? The vision of Nature of a given period depends on the imaginary which predominates during that period; in turn, that vision depends on a multiplicity of parameters: the degree of development of science and technology, social organization, art, religion, etc. Once formed, an image of Nature exercises an influence on all areas of knowledge. The passage from one vision to another is not progressive, continuous – it occurs by means of sharp, radical, discontinuous ruptures. Several contradictory visions can co-exist. The extraordinary diversity of visions of Nature explains why one cannot speak of Nature, but only of a certain nature in accord with the imaginary of a given period.

The image of Nature has always had a multiform action: it has influenced not only science but also art, religion, and social life. This allows us to explain some strange synchronicities. Here I limit myself to but a single example: the simultaneous appearance of the theory of the end of history and of the end of science just before the beginning of the 3rd millenium. For example, unified theories in physics have as their aim the elaboration of a complete approach, founded on a unique interaction, which can predict everything (hence the name, "Theory of Everything"). It is quite obvious that if such a theory were formulated in the future, it would signify the end of fundamental physics, because there would be nothing left to look for. It is interesting to observe that both the idea of the end of history and of the end of science have simultaneously emerged from the "end of the century" imaginary.

Notwithstanding the abundant and fascinating diversity of images of Nature one can nevertheless distinguish three main stages: Magic Nature, Nature as Machine, and the Death of Nature. Magical thought views nature as a living organism, endowed with intelligence and consciousness. The fundamental postulate of magical thought is that of universal interdependence: Nature cannot be conceived outside of its relations with us. Everything is sign, trace, signature, symbol. Science, in the modern sense of this word, is superfluous.

At the other extreme, the mechanist and determinist thought of the 18th and above all the 19th century (which, by the way, still predominates today) conceives Nature not as an organism, but as a machine. It suffices to disassemble this machine piece by piece in order to possess it entirely. The fundamental postulate of mechanistic and determinist thought is that Nature can be known and conquered by scientific methodology, defined in a way which is completely independent of human beings and separate from us.

The logical outcome of the mechanist and determinist vision is the Death of Nature, the disappearance of the concept of Nature from the scientific field. From the very beginning of the mechanistic vision, Nature as Machine, with or without the image of God as watchmaker, is split up into an ensemble of separate parts. From that moment on, there is no more need for a coherent whole, for a living organism, or even, for a machine which still kept the musty odor of finality. Nature is dead, but complexity remains. An astonishing complexity (in fact, often confused with "complication"), which penetrates each and every field of knowledge. But this complexity is perceived as an accident; we ourselves are considered to be an accident of complexity.

The Death of Nature is incompatible with a coherent interpretation of the results of contemporary science, in spite of the persistence of the neo-reductionistic attitude which accords exclusive importance to the fundamental building-blocks of matter and to the four known physical interactions. According to this neo-reductionist attitude, all recourse to Nature is superfluous and devoid of sense. In truth, Nature is dead only for a certain vision of the world – the classical vision.

The rigid objectivity of classical thought is only viable in the classical world. The idea of total separation between an observer and a Reality assumed to be completely independent from that observer brings us to the verge of insurmountable paradoxes. In fact, a far more subtle notion of objectivity characterizes the quantum world: objectivity depends on the level of Reality in question.

Space-time itself no longer rests on a fixed concept. Our space-time which proceeds in four dimensions is not the only conceivable space-time. According to certain physical theories, it appears more an approximation, like a part of a space-time all the more rich for being the generator of possible phenomena. Supplementary dimensions are not the result of mere intellectual speculation. On the one hand, these dimensions are necessary to insure the self-consistency of the theory and the elimination of certain undesirable aspects. On the other hand, they do not have a purely formal character – they have physical consequences for our own scale. For example, according to certain cosmological theories, if the universe had been associated from the "beginning" of the big bang in a multi-dimensional space-time, supplementary dimensions would have remained forever hidden, unobservable; rather, their vestiges would be precisely the known physical interactions. By means of generalizing the example provided by particle physics, it becomes conceivable that certain levels of Reality correspond to a space-time different than that characterizing our own level. Moreover, complexity itself would depend on the nature of space-time as well.

We can make, like Heisenberg made , a step further and assert that the classical four-dimensional space-time is, in fact, an *anthropomorphic concept*, founded on our sense-organs.

According to present scientific conceptions, *matter* is far from being identical with substance. In the quantum world, matter is associated with a *substance-energy-information-space-time complexus*.

It is somewhat mysterious why trajectories played such a central role in the formulation of modern physics. The quantum indeterminacy showed that trajectories are not a fondamental concept. In more recent years, a new discipline is born by the unexpected encounter between the theory of information and quantum mechanics: the Quantum Theory of Information . This new-born science already poses a crucial question : are the *information laws* more general, and therefore deeper, than the equations of movement? Are the central concepts of positions, speeds and trajectories of particles to be abandoned in favour of information laws which, in fact, could be valid not only for physics but also for other fields of knowledge? There were these last years fabulous experimental advances in the fields of non-separability, disentaglement, quantum cryptography and teleportation, in conjonction with the possible advent of quantum computers. This shows that notions like "levels of Reality" or "included middle" cease to be just theoretical speculations, by entering today in the field of experiments and, tomorrow, in the everyday life.

We can assert that the notion itself of *laws of Nature* completely changes its contents when compared with that of the classical vision. This situation can be summed up by three theses formulated by the well-known physicist Walter Thirring :

1\. *The laws of any inferior level are not completely determined by the laws of a superior level*. Thus, notions well anchored in classical physics, like "fundamental" and "accidental," must be re-examined. That which is considered to be fundamental on one level can appear to be accidental on a superior level and that which is considered to be accidental or incomprehensible on a certain level can appear to be fundamental on a superior level.

2\. *The laws of an inferior level depend more on the circumstances of their emergence than on the laws of a superior level*. The laws of a certain level depend essentially on the local configuration to which these laws refer. There is therefore a kind of local autonomy of respective levels of Reality; however, certain internal ambiguities concerning laws of an inferior level of Reality are resolved by taking into account the laws of a superior level. It is the internal consistency of laws which reduces the ambiguity of laws.

3\. *The hierarchy of laws evolves at the same time as the universe itself*. In other words, the *birth of laws* occurs simultaneously with the evolution of the universe. These laws pre-exist at the "beginning" of the universe as potentialities. It is the evolution of the universe which actualizes these laws and their hierarchy. A transdisciplinary model of Nature must integrate all this new knowledge of the emergent characteristics of the physical universe.

The Thirring's description of the laws of Nature is in perfect agreement with our own considerations about the Gödelian structure of Nature and knowledge. The problem of quantum indeterminacy can now be fully understood as *the influence of the quantum level of Reality on our own macrophysical level of Reality*. Of course, the laws of the macrophysical level depend more, as Thirring writes, on "the circumstances of their emergence". From the point of view of the macrophysical level indeterminacy appears as accidental, incomprehensible, or at most as a rare event. But this reveals, in fact, an internal ambiguity which can be solved only by taking into account the laws of the quantum level. At this last level the indeterminacy is fundamental.

One can ask if one can not logically conceive a *generalized indeterminacy*, which goes far beyond the problem of trajectories of particles. Heisenberg already considered the *indeterminacy of language* : the natural language can not express with arbitrary high precision all its elements, because the way of expressing acts in an essential manner on what is expressed. The indeterminacy of the natural language is just one example of the generalized indeterminacy generated by the Gödelian structure of Nature and knowledge.

In conclusion, we can distinguish three major aspects of Nature in accordance with the transdisciplinary model of Reality :

\(1\) *Objective Nature*, which is connected with the natural properties of the transdisciplinary Object; objective Nature is subject to subjective objectivity. This objectivity is subjective to the extent that the levels of Reality are connected to levels of perception. Nevertheless emphasis here is on objectivity, to the extent to which the methodology employed is that of science.

2\) *Subjective Nature*, which is connected with the natural properties of the transdisciplinary Subject; subjective Nature is subject to objective subjectivity. This subjectivity is objective to the extent that the levels of perception are connected to levels of Reality. Nevertheless, emphasis here is on subjectivity, to the extent to which the methodology is employed is that of the ancient science of being, which crosses all the traditions and religions of the world.

3\) *Trans-Nature*, which is connected with a similarity in Nature which exists between the transdisciplinary Object and the transdisciplinary Subject. Trans-Nature concerns the domain of the sacred. It cannot be approached without considering the other two aspects of Nature at the same time.

Transdisciplinary Nature has a ternary structure (objective Nature, subjective Nature, trans-Nature), which defines *living Nature*. This Nature is living because it is there that life is present in all its degrees and because its study demands the integration of *lived experience*. The three aspects of Nature must be considered simultaneously in terms of their inter-relation and their conjunction within all the phenomena of living Nature .

The study of living Nature asks for a new methodology – transdisciplinary methodology – which is different from the methodology of modern science and the methodology of the ancient science of being. It is the *co-evolution* of the human being and of the universe which asks for a new methodology.

An attempt to elaborate a new *Philosophy of Nature*, a privileged mediator of a dialogue between all the areas of knowledge, is one of the highest priorities of transdisciplinarity.

[^1]: Published in *Determinismo e complessità*, Fondazione Nova Spes and Armando Editore, Roma, 2000, pp. 127-158, edited by F. Tito Arecchi.

[^2]: Unité de Recherche des Universités Paris 6 et Paris 7, Associée au CNRS.


author: A. J. Landahl; D. S. Lobser; B. C. A. Morrison; K. M. Rudinger; A. E. Russo; J. W. Van Der Wall; P. Maunz
date:
  March 4, 2020
title:
  Jaqal™, the Quantum Assembly Language for QSCOUT

# To Learn More

To learn more about QSCOUT and the Jaqal™ language developed for it, please visit [qscout.sandia.gov](https://qscout.sandia.gov) or send an e-mail to <email@example.com>.

# Introduction

QSCOUT is the Quantum Scientific Computing Open User Testbed, a trapped-ion quantum computer testbed realized at Sandia National Laboratories on behalf of the Department of Energy's Office of Science and its Advanced Scientific Computing Research (ASCR) program. As an open user testbed, QSCOUT provides the following to its users:

-   **Transparency**: Full implementation specifications of the underlying native trapped-ion quantum gates.

-   **Extensibility**: Pulse definitions can be programmed to generate custom trapped-ion gates.

-   **Schedulability**: Users have full control of sequential and parallel execution of quantum gates.

## QSCOUT Hardware 1.0

The first version (1.0) of the QSCOUT hardware realizes a single register of qubits stored in the hyperfine clock states of trapped ${}^{171}$Yb${}^+$ ions arranged in a one-dimensional chain. Single and multi-qubit gates are realized by tightly focused laser beams that can address individual ions. The native operations available on this hardware include the following:

-   Global preparation and measurement of all qubits in the $z$ basis.

-   Parallel single-qubit rotations about any axis in the equatorial plane of the Bloch sphere.

-   The Mølmer–Sørensen two-qubit gate between any pair of qubits, in parallel with no other gates.

-   Single-qubit $Z$ gates executed virtually by adjusting the reference clocks of individual qubits.

Importantly, QSCOUT 1.0 does not support measurement of a subset of the qubits. Consequently, it also does not support classical feedback. This is because, for ions in a single chain, the resonance fluorescence measurement process destroys the quantum states of all qubits in the ion chain, so that there are no quantum states onto which feedback can be applied. Future versions of the QSCOUT hardware will support feedback.

QSCOUT 1.0 uses ***Just Another Quantum Assembly Language (Jaqal)*** (described [below](#jaqal-quantum-assembly-language)) to specify quantum programs executed on the testbed. On QSCOUT 1.0, every quantum computation starts with preparation of the quantum state of the entire qubit register in the $z$ basis. Then it executes a sequence of parallel and sequential single and two-qubit gates. After this, it executes a simultaneous measurement of all qubits in the $z$ basis, returning the result as a binary string. This sequence of prepare-all/do-gates/measure-all can be repeated multiple times in a Jaqal program, if desired. However, any adaptive program that uses the results of one such sequence to issue a subsequent sequence must be done with metaprogramming, because Jaqal does not currently support feedback. Once the QSCOUT platform supports classical feedback, Jaqal will be extended to support it as well.

# Gate Pulse File

The laser pulses that implement built-in or custom trapped-ion gates are defined in a ***Gate Pulse File (GPF)***. Eventually, users will be able to write their own GPF files, but that capability will not be available in our initial software release. However, users will be free to specify composite gates by defining them as sub-circuit [macros](#macro-statement). Additionally, custom native gates can be added in collaboration with Sandia scientists by specifying the pulse sequences that have to be applied to the trapped ion qubits to realize the gate.

We have provided a GPF file for the built-in gates on the QSCOUT 1.0 platform. This file is not intended to be modified by users, so we are not specifying its contents here. However, a full specification of the built-in gates will be available to users of the QSCOUT 1.0 platform. This GPF file contains pulse-level gate definitions for the QSCOUT 1.0 built-in gates listed below. All angle arguments in this list are in the units of radians, with 40 bits of precision. The chirality of rotations is determined using the right-hand rule.

-   `prepare_all`
    0.875cm Prepares all qubits in the quantum register in the $|0\rangle$ state in the $z$ basis.

-   `R <qubit> <axis angle> <rotation angle>`
    Counter-clockwise rotation around an axis in the equatorial plane of the Bloch sphere defined by `<axis angle>`, measured counter-clockwise from the $x$ axis, by the angle defined by `<rotation angle>`.

-   `Rx <qubit> <rotation angle>`
    Counter-clockwise rotation around the $x$ axis, by the angle defined by `<rotation angle>`.

-   `Ry <qubit> <rotation angle>`
    Counter-clockwise rotation around the $y$ axis, by the angle defined by `<rotation angle>`.

-   `Rz <qubit> <angle>`
    Counter-clockwise rotation around the $z$ axis, by the angle defined by `<rotation angle>`.

-   `Px <qubit>`
    Counter-clockwise rotation around the $x$ axis, by $\pi$. (Pauli $X$ gate.)

-   `Py <qubit>`
    Counter-clockwise rotation around the $y$ axis, by $\pi$. (Pauli $Y$ gate.)

-   `Pz <qubit>`
    Counter-clockwise rotation around the $z$ axis, by $\pi$. (Pauli $Z$ gate.)

-   `Sx <qubit>`
    Counter-clockwise rotation around the $x$ axis, by $\pi/2$. ($\sqrt{X}$ gate.)

-   `Sy <qubit>`
    Counter-clockwise rotation around the $y$ axis, by $\pi/2$. ($\sqrt{Y}$ gate.)

-   `Sz <qubit>`
    Counter-clockwise rotation around the $z$ axis, by $\pi/2$. ($\sqrt{Z}$ gate.)

-   `Sxd <qubit>`
    Clockwise rotation around the $x$ axis, by $\pi/2$. ($\sqrt{X}^\dagger$ gate.)

-   `Syd <qubit>`
    Clockwise rotation around the $y$ axis, by $\pi/2$. ($\sqrt{Y}^\dagger$ gate.)

-   `Szd <qubit>`
    Clockwise rotation around the $z$ axis, by $\pi/2$. ($\sqrt{Z}^\dagger$ gate.)

-   `MS <qubit> <qubit> <axis angle> <rotation angle>`
    The general two-qubit Mølmer–Sørensen gate. (If we let $\theta$ represent `<rotation angle>` and $\varphi$ represent `<axis angle>`, then the gate is $$\exp\left(-i\left(\frac{\theta}{2}\right)(\cos \varphi X + \sin \varphi Y)^{\otimes 2}\right).$$

-   `Sxx <qubit> <qubit>`
    The XX-type two-qubit Mølmer–Sørensen gate: $$\exp\left(-i\left(\frac{\pi}{2}\right) X\otimes X\right).$$

-   `measure_all`
    Measures all qubits of the quantum register in the $z$ basis. After measurement, ions will be outside the qubit space. Therefore, the qubits have to be prepared again before any other gates can be applied.

The gate pulse definitions also include idle gates with the same duration as the single- and two-qubit gates. These have a prefix of `I_`. For example an idle gate of the same duration as a `Px` can be obtained by `I_Px <qubit>`. It is important to note that it is not necessary to explicitly insert idle on idling qubits in a parallel block. Explicit idle gates are meant to be used for performance testing and evaluation.

The open nature of the QSCOUT testbed requires a flexible **Quantum Assembly Language (QASM)** that empowers QSCOUT users to extend the set of native gates and fully control the execution of the quantum program on the QSCOUT testbed. Due to the proliferation of such languages in this fledgling field, ours is named **Just Another Quantum Assembly Language**, or **Jaqal**.

To realize our objectives, the Jaqal QASM language fulfills the following requirements:

-   Jaqal fully specifies the allocation of qubits within the quantum register, which *cannot* be altered during execution.

-   Jaqal requires the scheduling of sequential and parallel gate sequencing to be fully and explicitly specified.

-   Jaqal can execute any native (built-in or custom) gate specified in any GPF file it references.

While Jaqal is built upon a lower-level pulse definition in GPF files, it is the lowest-level QASM programming language exposed to users in QSCOUT. We anticipate that users will develop their own higher-level programming languages that compile down to Jaqal. We plan to release Jaqal-branded metaprogramming tools after user-driven innovation at this meta-programming level settles down.

# Jaqal Syntax

A Jaqal file consists of gates and metadata making those gates easier to read and write. The gates that are run on the machine can be deterministically computed by inspection of the source text. This implies that there are no conditional statements at this level. This section will describe the workings of each statement type.

Whitespace is largely unimportant except as a separator between statements and their elements. If it is desirable to put two statements on the same line, a ';' separator may be used. In a parallel block, the pipe ('') must be used instead of the ';'. Like the semicolon, however, the pipe is unnecessary to delimit statements on different lines. Both Windows and Linux newline styles will be accepted.

## Identifiers

Gate names and qubit names have the same character restrictions. Similar to most programming languages, they may contain, but not start with, numerals. They are case sensitive and may contain any non-accented Latin character plus the underscore. Identifiers cannot be any of the keywords of the language.

## Comments

C/C++ style comments are allowed and treated as whitespace. A comment starting with '//' runs to the end of the current line, while a comment with '/\*' runs until a '\*/' is encountered. These comments do not nest, which is the same behavior as C/C++.

## Header Statements

A properly formatted Jaqal file comprises a header and body section. All header statements must precede all body statements. The order of header statements is otherwise arbitrary except that all objects must be defined before their first use.

### Register Statement

A register statement serves to declare the user's intention to use a certain number of qubits, referred to in the file with a given name. If the machine cannot supply this number of qubits then the entire program is rejected immediately.

The following line declares a register named `q` which holds 7 qubits.

    register q[7]

### Map Statement

While it is sufficient to refer to qubits by their offset in a single register, it is more convenient to assign names to individual qubits. The map statement effectively provides an alias to a qubit or array of qubits under a different name. The following lines declare the single qubit `q``[``0``]` to have the name `ancilla` and the array `qubits` to be an alias for `q`. Array indices start with 0.

    register q[3]
    map ancilla q[0]
    map qubits q

The map statement will also support Python-style slicing. In this case, the map statement always declares an array alias. In the following line we relabel every other qubit to be an ancilla qubit, starting with index 1.

    register q[7]
    map ancilla q[1:7:2]

After this instruction, `ancilla``[``0``]` corresponds to `q``[``1``]`; `ancilla``[``1``]` and `ancilla``[``2``]` correspond to `q``[``3``]`and `q``[``5``]`, respectively.

### Let Statement

We allow identifiers to replace integers or floating point numbers for convenience. There are no restrictions on capitalization. An integer defined in this way may be used in any context where an integer literal is valid and a floating point may similarly be used in any context where a floating point literal is valid. Note that the values are constant, once defined.

Example:

    let total_count 4
    let rotations 1.5

## Body Statements

### Gate Statement

Gates are listed, one per statement, meaning it is terminated either by a newline or a separator. The first element of the statement is the gate name followed by the gate's arguments which are whitespace-separated numbers or qubits. Elements of quantum registers, mapped aliases, and local variables (see section on [macros](#macro-statement)) may be freely interchanged as qubit arguments to each gate. The names of the gates are fixed but determined in the Gate Pulse File, except for macros. The number of arguments ("arity") must match the expected number. The following is an example of what a 2-qubit gate may look like.

    register q[3]
    map ancilla q[1]
    Sxx q[0] ancilla

The invocation of a macro is treated as completely equivalent to a gate statement.

### Gate Block

Multiple gates and/or macro invocations may be combined into a single block. This is similar, but not completely identical, to how C or related languages handle statement blocks. Macro definitions and header statements are not allowed in gate blocks. Additionally, statements such as macro definitions or loops expect a gate block syntactically and are not satisfied with a single gate, unlike C.

Two different gate blocks exist: sequential and parallel. Sequential gate blocks use the standard C-style '{}' brackets while parallel blocks use angled '\<\>' brackets, similar to C++ templates. This choice was made to not conflict with '\[\]' brackets, which are used in arrays, and to reserve '()' for possible future use. In a sequential block, each statement, macro, or gate block waits for the previous to finish before executing. In a parallel gate block, all operations are executed at the same time. It is an error to request parallel operations that the machine is incapable of performing, however it is not syntactically possible to forbid these as they are determined by hardware constraints which may change with time.

[Looping statements](#loop-statement) are allowed inside sequential blocks, but not inside parallel blocks. Blocks may be arbitrarily nested so long as the hardware can support the resulting sequence of operations. Blocks may not be nested directly within other blocks of the same type.

The following statement declares a parallel block with two gates.

    < Sx q[0] | Sy q[1] >

This does the same but on different lines.

    <
        Sx q[0]
        Sy q[1]
    >

Here is a parallel block nested inside a sequential one.

    {
        Sxx q[0] q[1]
        < Sx q[0] | Sy q[1] >
    }

And sequential blocks may be nested inside parallel blocks.

    <
        Sx q[0]
        { Sx q[1] ; Sy q[1] }
    >

### Timing within a parallel block

If two gates are in a parallel block but have different durations (*e.g.*, two single-qubit gates of different length), the default behavior is to *start* each gate within the parallel block simultaneously. The shorter gate(s) will then be padded with idles until the end of the gate block. For example, the command

    <
        Rx q[1] 0.1
        Sx q[2]
    >

results in the `Rx` gate on `q``[``1``]` with angle 0.1 radians and `Sx` gate on `q``[``2``]` both starting at the same time; the `Rx` gate will finish first and `q``[``1``]` will idle while the `Sx` gate finishes. Once the Jaqal gate set becomes user-extensible, users may define their own scheduling within parallel blocks (*e.g.*, so that gates all *finish* at the same time instead).

A macro can be used to treat a sequence of gates as a single gate. Gates inside a macro can access the same qubit registers and mapped aliases at the global level as all other gates, and additionally have zero or more arguments which are visible. Arguments allow the same macro to be applied on different combinations of physical qubits, much like a function in a classical programming language.

A macro may use other macros that have already been declared. A macro declaration is complete at the *end* of its code block. This implies that recursion is impossible. It also implies that macros can only reference other macros created earlier in the file. Due to the lack of conditional statements, recursion always creates an infinite loop and is therefore never desirable.

A macro is declared using the `macro` keyword, followed by the name of the macro, zero or more arguments, and a code block. Unlike C, a macro must use a code block, even if it only has a single statement.

The following example declares a macro.

    macro foo a b {
        Sx a
        Sxx a q[0]
        Sxx b q[0]
    }

To simplify parsing, a line break is not allowed before the initial '{', unlike C. However, statements may be placed on the same line following the '{'.

A gate block may be executed for a fixed number of repetitions using the loop statement. The loop statement is intentionally restricted to running for a fixed number of iterations. This ensures it is easy to deterministically evaluate the runtime of a program. Consequently, it is impossible to write a program which will not terminate.

The following loop executes a sequence of statements seven times.

    loop 7 {
        Sx q[0]
        Sz q[1]
        Sxx q[0] q[1]
    }

The same rules apply as in macro definitions: '{' must appear on the same line as `loop`, but other statements may follow on the same line.

Loops may appear in sequential gate blocks, but not in parallel gate blocks.

# Extensibility

As Jaqal and the QSCOUT project more broadly have extensibility as stated goals, it is important to clarify what is meant by this term. Primarily, Jaqal offers extensibility in the gates that can be performed. This will occur through the gate pulse file and the use of macros to define composite gates that can be used in all contexts a native gate can. Jaqal will be incrementally improved as new hardware capabilities come online and real world use identifies areas for enhancement. The language itself, however, is not intended to have many forms of user-created extensibility as a software developer might envision the term. Features we do not intend to support include, but are not limited to, pragma statements, user-defined syntax, and a foreign function interface (i.e. using custom C or Verilog code in a Jaqal file).

# Examples

## Bell state preparation

This example prepares a Bell state using the classic Hadamard and controlled X circuit, then measures it in the computational basis. Up to the limits of gate fidelity, the measurements of the two qubits should always match.

    macro hadamard target { // A Hadamard gate can be implemented as
        Sy target           // a pi/2 rotation around Y
        Px target           // followed by a pi rotation around X.
    }

    macro cnot control target {  // CNOT implementation from Maslov (2017)
        Sy control               //
        Sxx control target
        <Sxd control | Sxd target>  // we can perform these in parallel
        Syd control
    }

    register q[2]

    prepare_all         // Prepare each qubit in the computational basis.
    hadamard q[0]
    cnot q[1] q[0]
    measure_all         // Measure each qubit and read out the results.

However, there's a more efficient way of preparing a Bell state that takes full advantage of the native Mølmer-Sørensen interaction of the architecture, rather than using it to replicate a controlled-X gate. The following snippet of code repeats that interaction 1024 times, measuring and resetting the ions after each time. All 1024 measurement results will be reported to the user.

    register q[2]

    loop 1024 {
        prepare_all
        Sxx q[0] q[1]
        measure_all
    }

## Single-Qubit Gate Set Tomography

    register q[1]

    // Fiducials
    macro F0 qubit { I_Sx qubit }
    macro F1 qubit { Sx qubit }
    macro F2 qubit { Sy qubit }
    macro F3 qubit { Sx qubit; Sy qubit}
    macro F4 qubit { Sx qubit; Sx qubit; Sx qubit }
    macro F5 qubit { Sy qubit; Sy qubit; Sy qubit }

    // Germs
    macro G0 qubit { Sx qubit }
    macro G1 qubit { Sy qubit }
    macro G2 qubit { I_Sx qubit }
    macro G3 qubit { Sx qubit; Sy qubit }
    macro G4 qubit { Sx qubit; Sy qubit; I_Sx qubit }
    macro G5 qubit { Sx qubit; I_Sx qubit; Sy qubit }
    macro G6 qubit { Sx qubit; I_Sx qubit; I_Sx qubit }
    macro G7 qubit { Sy qubit; I_Sx qubit; I_Sx qubit }
    macro G8 qubit { Sx qubit; Sx qubit; I_Sx qubit; Sy qubit }
    macro G9 qubit { Sx qubit; Sy qubit; Sy qubit; I_Sx qubit }
    macro G10 qubit { Sx qubit; Sx qubit; Sy qubit; Sx qubit; Sy qubit; Sy qubit }

    // Length 1
    prepare_all
    F0 q[0]
    measure_all

    prepare_all
    F1 q[0]
    measure_all

    prepare_all
    F2 q[0]
    measure_all

    prepare_all
    F3 q[0]
    measure_all

    prepare_all
    F4 q[0]
    measure_all

    prepare_all
    F5 q[0]
    measure_all

    prepare_all
    F1 q[0]; F1 q[0]
    measure_all

    prepare_all
    F1 q[0]; F2 q[0]
    measure_all

    // and many more
    // Repeated germs can be realized with the loop

    prepare_all
    F1 q[0]
    loop 8 { G1 q[0] }
    F1 q[0]
    measure_all

When successfully executed, a single Jaqal file will generate a single ASCII text file (Linux line endings) in the following way:

1.  Each call of `measure_all` at runtime will add a new line of data to the output file. (If `measure_all` occurs within a `loop` (or nested loops), then multiple lines of data will be written to the output file, one for each call of `measure_all` during execution.)

2.  Each line of data written to file will be a single bitstring, equal in length to the positive integer passed to `register` at the start of the program.

3.  Each bitstring will be written in least-significant bit order (little endian).

For example, consider the program:

    register q[2]

    loop 2 {
        prepare_all
        Px q[0]
        measure_all
    }

    loop 2 {
        prepare_all
        Px q[1]
        measure_all
    }

Assuming perfect execution, the output file would read as:

    10
    10
    01
    01

While this output format will be "human-readable", it may nevertheless be unwieldy to work with directly. Therefore, a Python-based parser will be written to aid users in manipulating output data.

# Possible Future Capabilities

Jaqal is still under development, and will gain new features as the QSCOUT hardware advances. While the precise feature set of future versions of Jaqal is still undetermined, we discuss some features that may be added, and in some cases identify workarounds for the current lack of those features.

## Subset Measurement

Currently, the measurement operation of the QSCOUT hardware acts on all ions in the trap, destroying their quantum state and taking them out of the computational subspace. Future versions of the QSCOUT hardware will allow for the isolation and measurement of a subset of qubits with a command of the form `measure_subset <qubit> ...`. Similarly, a `prepare_subset <qubit> ...` operation will allow the reuse of measured qubits without destroying the quantum state of the remainder. These would be implemented in a Gate Pulse File, and not require a change to the Jaqal language.

## Measurement Feedback

The QSCOUT hardware does not currently support using measurement outcomes to conditionally execute future gates. We expect this capability will be added in a future version of the QSCOUT hardware, and Jaqal programs will be able to use that capability once it exists. We have chosen to delay adding the syntax for measurement feedback to Jaqal until that time, in order to allow us the flexibility to choose a syntax that best allows users to take advantage of the actual capabilities of our hardware, once those are known.

## Classical Computation

Jaqal does not currently support any form of classical computation. We understand that this is a limitation, and expect future versions of Jaqal to do so. There are two relevant forms of classical computation that we are considering for Jaqal.

### Compile-Time Classical Computation

Performing classical computations at compile-time, before the program is sent to the quantum computer, can vastly increase the expressiveness of the language. For example, consider the following experiment, *which is not currently legal Jaqal code:*

    register q[1]

    let pi 3.1415926536

    loop 100 {
        prepare_all
        Ry q[0] pi/32
        measure_all
        prepare_all
        Ry q[0] pi/16
        measure_all
        prepare_all
        Ry q[0] 3*pi/32
        measure_all
        prepare_all
        Ry q[0] pi/8
        measure_all
    }

Currently, Jaqal does not support inline parameter calculations like the above. The recommended workaround is to define additional constants as needed:

    register q[1]

    let pi_32   0.09817477042
    let pi_16   0.1963495408
    let pi_3_32 0.2945243113
    let pi_8    0.3926990817

    loop 100 {
        prepare_all
        Ry q[0] pi_32
        measure_all
        prepare_all
        Ry q[0] pi_16
        measure_all
        prepare_all
        Ry q[0] pi_3_32
        measure_all
        prepare_all
        Ry q[0] pi_8
        measure_all
    }

Another example of a case where compile-time classical computation could be useful is in macro definition. For example, if you wished to define a macro for a controlled z rotation in terms of a (previously-defined) CNOT macro:

    ...
    macro CNOT control target { ... }

    macro CRz control target angle {
        Rz target angle/2
        CNOT control target
        Rz target -angle/2
        CNOT control target
    }
    ...

Again, the above example *is not currently legal Jaqal.* We recommend, in such cases, that you manually unroll macros as needed, then define additional constants as above. That is, rather than using the above macro:

    ...
    let phi 0.7853981634;
    ...
    CRz q[0] q[1] phi;
    ...

You should instead call the gates the macro is made up of, substituting the results of the appropriate calculations yourself:

    ...
    let phi 0.7853981634;
    let phi_2 0.3926990817;
    let phi_m_2 -0.3926990817;
    ...
    Rz q[1] phi_2; CNOT q[0] q[1]; Rz q[1] phi_m_2; RNOT q[0] q[1];
    ...

We recognize that this "manual compilation" is a significant inconvenience for writing readable and expressive code in Jaqal. We expect to include compile-time classical computation in a relatively early update to Jaqal, likely even before measurement feedback is available. Fortunately, metaprogramming (automated code generation) significantly eases the burden of the lack of classical computation features, and we highly recommend it to users of Jaqal.

### Run-Time Classical Computation

Users may also wish to do classical computation while a Jaqal program is running, based on the results of measurements. For example, in hybrid variational algorithms, a classical optimizer may use measurement results from one circuit to choose rotation angles used in the next circuit. In error-correction experiments, a decoder may need to compute which gates are necessary to restore a state based on the results of stabilizer measurements. Adaptive tomography protocols may need to perform statistical analyses on measurement results to determine which measurements will give the most information. As can be seen from the above examples, run-time classical computation is useful only when measurement feedback is possible. Accordingly, we will consider this feature after we have added support for measurement feedback. However, use cases like adaptive tomography and variational algorithms can be implemented via metaprogramming techniques. After running a Jaqal file on the QSCOUT hardware, a metaprogram can parse the [measurement results](#data-output-format), then use that information to generate a new Jaqal file to run.

## Randomness

Executing quantum programs with gates chosen via classical randomness is desirable for a variety of reasons. Applications of randomized quantum programs include hardware benchmarking, error mitigation, and some quantum simulation algorithms. Jaqal does not currently have built-in support for randomization, although it may in the future, likely in combination with support for run-time classical computation. Our currently recommended workaround is to pre-compute any randomized elements of the algorithm, automatically generating Jaqal code to execute the random circuit selected. For example, the following program isn't currently possible, as there's no means of generating a random angle in Jaqal directly:

    register q[1]

    loop 100 {
        prepare_all
        // Do an X rotation on q[0] by a random angle between 0 and 2*pi.
        measure_all
    }

However, the same effect can be obtained by a metaprogram (written in Python, for the sake of example) that generates a Jaqal program:

    from random import uniform
    from math import pi
    with open("randomness_example.jql", "w") as f:
        f.write("register q[1]\n\n")
        for idx in range(100):
            angle = uniform(0.0, 2.0 * pi)
            f.write("prepare_all\n")
            f.write("Rx q[0] %f\n" % angle)
            f.write("measure_all\n\n")

While the generated Jaqal program is much larger than one that could be written in a potential future version of Jaqal that supported randomized execution, the metaprogram that generates it is quite compact.

# Acknowledgements

This material was funded by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research Quantum Testbed Program.

Sandia National Laboratories is a multi-mission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for DOE's National Nuclear Security Administration under contract DE-NA0003525.


abstract: The HANDE quantum Monte Carlo project offers accessible stochastic algorithms for general use for scientists in the field of quantum chemistry. HANDE is an ambitious and general high-performance code developed by a geographically-dispersed team with a variety of backgrounds in computational science. In the course of preparing a public, open-source release, we have taken this opportunity to step back and look at what we have done and what we hope to do in the future. We pay particular attention to development processes, the approach taken to train students joining the project, and how a flat hierarchical structure aids communication.
author: J. S. Spencer; N. S. Blunt; W. A. Vigor; Fionn D. Malone; W. M. C. Foulkes; James J. Shepherd; A. J. W. Thom
bibliography: hande_wssspe2.bib
date: 2024-09-30
title: Open-source development experiences in scientific software: the HANDE quantum Monte Carlo project

[^1]

[^2]

The Highly Accurate N-DEterminant (HANDE) quantum Monte Carlo project began life as an experiment by one of us (JSS) to explore the (then recent) development in quantum chemistry: the full configuration interaction quantum Monte Carlo (FCIQMC) method . FCIQMC can be viewed simply as a stochastic approach to the power method; it allows the calculation of exact ground state energies of quantum systems with Hilbert spaces orders of magnitude larger than accessible via even state-of-the-art deterministic algorithms. Initially only the Hubbard model was implemented, but HANDE now handles a range of model and chemical systems. At the same time HANDE has become an efficient and highly parallel implementation of FCIQMC and related methods, capable of scaling to several thousand cores. We have also provided deeper understanding of the FCIQMC method, extended HANDE to include the canonical implementation of the stochastic coupled cluster approach and developed new methods within the field. The driving-force for this transformation, from a toy code to a professional software package, has been the team of contributors split between three universities working together in a sustainable and robust process. We are very proud of the variety of our developers, who represent several different areas of science and range from undergraduates to professors. Indeed, we have had exceptional success with undergraduate research projects, which is remarkable given that most start with no or little experience in parallel computing and in quantum chemistry—a notable example is the development of a novel Monte Carlo method by two undergraduate students.

The unexpected and organic growth has provided its challenges. How to transition into a community-owned code from the initial gatekeeper model we stumbled into? How to develop and support new contributors to the project? In some cases we planned ahead; in others we reached a consensus through iterative experimentation. Indeed, we have found flexibility and willingness to adapt to be of vital importance.

In this contribution we first describe the choices we made in an effort to write a sustainable, portable library, the approach we have settled on for development and the benefits we have subsequently obtained. We then discuss how we have trained students to be successful and valuable members of the development team and our future plans for the HANDE project before offering our conclusions and suggestions to the wider computational science community.

# HANDE OVERVIEW

HANDE is a small, but growing, project with half a dozen active developers at any one time. Most users are also developers but the user community is growing through active collaborations. The code base contains approximately 20000 lines of Fortran 2003, plus a smaller amount of C and several thousand lines of comments and is parallelised using MPI and OpenMP. HANDE is available as a source distribution via the project website and github. The distribution also contains a substantial amount of documentation, including compilation and usage instructions, and tutorials as well as python modules for data analysis. HANDE is developed on Linux, Mac OS X and Windows though, due to the nature of supercomputers, production calculations on high performance computer facilities are universally performed on Linux.

# A DEVELOPMENT MODEL

We view ourselves as scientists *and* programmers (though our funding agencies might not agree!) and believe both roles are vital. As programmers, a maintainable and efficient code is our main goal. As scientists, we wish to rapidly address the questions posed in our research. These positions are not, however, contradictory: rather we have found the programmers' goal also minimizes delays in making scientific progress once spread over a number of consecutive projects. In other words, poor design and development choices eventually hinder us. Here we detail some of the choices we have made and their consequences. We note that the comments we have to make are surprisingly general; an in-depth knowledge of the algorithm is not necessary to appreciate what we are discussing. We are, however, aided by FCIQMC and related methods being simple and composed of only a few distinct data flows. In particular, the memory demands are dominated by the representation of the eigenvector and the computational cost per iteration by the tight loop in which the eigenvector is stochastically evolved.

**Coding conventions**— We have taken care to maintain consistency in coding conventions throughout. This begins with a common, ordered commenting style; this visual cue helps developers become immediately aware of the existence of code norms and leads to it being easier to maintain wide-spread adoption of the other features below. Apart from making it far easier and more pleasant to read and understand code, such conventions serve as a guide to those with little prior experience programming and help prevent code from being rushed. We ensure that the functionality and inputs and outputs of all procedure interfaces are documented; this can then be extracted using tools such as sphinx and makes comprehension whilst navigating code (e.g. using ctags) far faster. We further advocate the use of extensive commenting to provide both an overview of the theory and the choices that lie behind an implementation: indeed, in the more theoretically challenging parts of HANDE, the amount of comments rivals or exceeds the actual amount of code. Such cases can be viewed as an example of literate programming and may include theoretical overviews (which, for research software, are frequently not yet available in the literature), a discussion on implementation choices, benchmarks, examples and so on. These serve both as documentation and as extremely helpful material from which new members of the development team can learn about details which may be inappropriate for traditional papers.

**Pure functions**— A growing trend in HANDE development, which has been successful, is a move towards the use of pure functions, which (along with other functional programming approaches) have been demonstrated to have compelling advantages. The results of pure functions depend only upon the input argument values and have no side effects on any part of the code outside the function. As such, pure functions cannot depend on any global data. We have found that functions which depend heavily on global data have many subtle interactions and assumptions, such that changing one part of the code can unexpectedly alter other parts. This problem becomes worse as the size of a program grows. In contrast, one can be confident that changes outside a pure function can never alter its results for the same set of inputs. Beyond this, code written in a pure style is more reusable (both within the code and in separate projects) and easier to test. Whilst writing code in a pure style can initially take longer, we are finding that it saves significant time and effort in the long run and makes implementing new functionality far easier. We have utilized this for threaded parallelism and alternate implementations.

**Factorisation**— Open source software provides a huge advantage to our developers; they are encouraged to extract code which could be reused in other projects to contribute to the community. This approach to factorisation forces developers to plan and separate functionally and logically independent code, improving the quality and sustainability of the code. Conversely we benefit from similar efforts in the broader community and can use state-of-art portable libraries to minimise time-to-science and avoid duplication of effort. For example, we use HDF5 for checkpoint files , dSFMT for random numbers  and the python scientific stack (especially numpy , pandas  and matplotlib ) for data analysis. In return, our contributions include Fortran interfaces to libraries , a test framework  (see below) and a python library for removing serial correlations in Monte Carlo data . We find such efforts are a way of broadening impact of our development work far beyond the immediate stochastic quantum chemistry community. Encouragingly, we have also received contributions to these libraries from outside of our team. Making the code publicly accessible via distributed version control (e.g. on github) is key to reducing the barrier to entry.

Despite the above, a large number of dependencies is undesirable from a usability viewpoint: requiring the user to manually compile several packages before using our program hinders experimentation and porting to new platforms. We try to overcome this in two ways: small libraries with permissive licenses can be included in the source distribution and non-core features which depend upon larger libraries can be disabled at compile-time.

**Pull requests and code review**— In the last year we have moved to a system of pull requests based upon the git flow model. In this system, any contributions to HANDE must be made on a branch (using our version control system of choice, git) and a review of the branch performed (by at least one other contributor) before it may be merged into master (see Fig. 1). Code review can easily be performed using (e.g.) github's inline commenting or, our preferred tool, watson. Code review is deliberately light weight and allows for rapid peer feedback about the approach used, problems in the design and consistency in code style. In particular, the process typically includes validation and verification, of the code, documentation and (crucially) any new theoretical work underlying it. We have found that this process greatly reduces bugs and rushed code from ending up in the master, which is designated to be sufficiently stable for production calculations. Already we have seen substantial improvements in the flexibility, sustainability and maintainability of the code. It also gives contributors an understanding of parts of the codebase that they may not otherwise know much about. Even those who do not perform a review in detail gain knowledge of the various projects being worked on. The social impact of this is interesting: we find code review to be an excellent way of flattening the academic hierarchical structure. In particular, we note that the levels of expertise in scientific and computational domains are often not aligned and the more 'junior' members of a research team are often the ones doing the most software development and hence their reviews of contributions from more 'senior' members can be the most enlightening.

One aspect deserves special consideration: not all development work is evolutionary; some must be revolutionary. This kind of development work is frequently long running and handling both the review and merging (often into a very different codebase after months of parallel development) is painful. We have found that regular peer review of intermediate work and occasional rebasing of such branches against the current development version of the code goes a long way to mitigating such issues.

**Regression testing**— Scientific codes produce quantitative results that, in principle, should be extremely simple to test against when the code changes. When differences happen to indicate a bug, these can be tracked down between a relatively small number of commits using a bisection method. Whilst unit tests are valuable, we have found that regression tests are easier to retrofit to existing code bases and are good at capturing problems in the interfaces between procedures or changes compared to existing answers. This type of regression testing is relatively straight-forward to undertake. Apart from data extraction from output files, regression testing involves a generic set of tasks. One of us (JSS) maintains an open source portable tool for just such a purpose , which has attracted use in the wider electronic structure community. Running the tests can be automated (e.g. to check every commit, every pull request, given time intervals) using tools such as jenkins, travis-ci or buildbot, which is currently used in the HANDE project, as is performed by many other projects (e.g. ). The design of tests themselves is a non-trivial challenge, and should not be underestimated. A test should check a broad sweep of functionality, but when there are many input parameters (and variably sparse matrices) it is impossible to check every combination, though tools such as gcov are invaluable in discovering the fraction of the code covered by a set of tests. HANDE contains over 160 tests which cover over 85% of the code base (excluding external libraries) and increasing this is an ongoing effort. Moreover, because the software is designed for high-performance computing and contains Monte Carlo algorithms, it can be hard to reliably review this functionality, especially for bugs which are only revealed when run on thousands of processors. Where possible, therefore, new conceptual developments are checked against numbers from other codes. A community which supports this kind of data sharing is extremely important for reliable scientific reproducibility.

**Reproducibility**— Reproducibility of experimental results is one of the most important principles in the scientific community. Numerical experiments should be held to as high standards, but often this is more difficult than it seems as code can change rapidly over time. This is even more problematic for Monte Carlo algorithms where newly introduced features can alter the Markov chain resulting in slightly different numerical answers. Furthermore, complex calculations rely upon an existing set of input and checkpoint files and produce similar numbers of files as output, making data provenance complicated. As a simple measure to overcome this we output the input options and the git commit hash to the main output file and a UUID specific to the calculation in all output files which enables us and any other user to reproduce the results of a particular calculation. We are fans of the IPython Notebook for data analysis as a way of storing the analysis and output together. These notebooks also represent useful training aids.

**Modern Standards**— Languages continue to evolve and exploiting new developments can be a powerful tool in making code more flexible, portable and maintainable. For example, the C interoperability features in Fortran 2003 make it much easier to combine existing code written in either language and so reduces the need to 'reinvent the wheel'. One word of caution: new language features are implemented at different rates across different compilers, which are updated infrequently in some environments. It is important to balance using new language features and staying away from the bleeding edge. Regular testing against a variety of common compilers is vital in maintaining the portability of the code.

**Bug fixing**— Bug fixing in an academic environment is somewhat fraught given the inherently fluctional development community. Whilst we have found the many bugs are prevented (or rather, discovered at time-of-creation) by code review, inevitably bugs remain to be discovered at a later time. Whilst debugging is a universally hard problem, especially (as is often the case in academia) when the original student or researcher has moved on, we have found the approaches we discussed above crucial in mitigating this factor. Good documentation, commenting and tests provide an indication of what the code should do (or at least what its author thought it should do!) and remove one layer of mystery. We have also found code review an excellent strategy to aid this; having multiple developers review and understand a section of the codebase (albeit perhaps not on the same level as its author) aids the spread of knowledge throughout the development team and helps make it more likely that at least one person is capable of fixing the bug relatively quickly. Once a bug is reported, it is triaged and a fix is proposed. Following our standard code review process, it is then merged into the stable branch. It is then important to update the test suite so that the bug remains fixed. Who does this work can be problematic, especially in cases where the original is no longer working on HANDE. Sometimes a code developer tracks the problem down. In other cases we find the open source adage of 'scratching your own itch' useful: the user who wants a bug fixed will (hopefully!) be suitably motivated to also fix it, given support and guidance from the wider development team. We have found that this can be a powerful tool for encouraging users to become developers.

# TRAINING

The challenges facing someone joining a computational science project are multi-faceted: one must be knowledgeable in broad technical issues, the programming language(s) used as well as the theory of the underlying science. However, in practice, applied computer science is often attempted in academia without formal training. This requires that students learn on-the-job, but students often come highly motivated to learn new skills from day one. Fortunately there are now excellent and affordable courses aimed at improving technical skills of computational scientists run by universities, national bodies (e.g. ARCHER in the UK) and international groups. We especially praise the impact of Software Carpentry.

**Introduction to HANDE**— Ideally, the instruction given should be: 'checkout the code and play around with it' and that should be sufficient; we aim for this to be the case. New developers frequently comment that strategies mentioned in the previous section greatly help them in coming to grips with the code and in keeping initial motivation high. We note this is a constant battle: additional features, optimisation and poor habits can cause the barrier of entry to creep up over time. However, we find a mindful approach beneficial. We recognise that initial impressions matter and so aim to make things as smooth as possible. We find that the speed at which new developers learn is helped by

Our experience is that highly-motivated students on moving away from the community willingly stay involved and enjoy doing so; this sets good examples for incoming students. Informal, nonhierarchical, peer-based management greatly enhances this effect; learning happens organically in an environment where asking questions is easy and group discussion common.

**Converting users to developers**— By the very nature of academia, the development community around research software fluctuates. Converting users into developers helps substantially in making a project sustainable, especially in niche fields. In addition to attempting to minimise the barrier to entry, we find a powerful technique is to encourage users to 'scratch their own itch': when a user has a feature request, we try to help them to implement it themselves (even if this takes more time than a core developer doing it themselves). The time investment is typically rewarded surprisingly quickly.

**Coding retreats**— Engendering a development community and sharing knowledge across a geographically dispersed network is hard. To this end we recently held a residential coding retreat. Those in attendance were encouraged to implement a simple feature (*i.e.* could be completed in the time available) of interest; coding review happened on-site. We found this to be a good community-building format. An important feature was to set aside substantial amounts of time for informal presentations and discussions, which provided a forum to discuss ongoing research as well as the codebase.

# DISCUSSION

We conclude with some examples of where our approach succeeded and where it failed, followed by an outlook on the future.

The development of a flexible, modular code supported by a training regime for new team members might appear to be a bet which may or may not pay off. Our experiences show that it does pay off; in fact many of the approaches we discussed above were suggested naturally and adopted due to frustration with inefficiencies from *not* doing them. The impact on our work has been tremendous. For example, two undergraduate students in a few months were able to propose, implement and test a new finite-temperature Monte Carlo approach in electronic structure. This would not have been possible if they had to start from scratch or from a monolithic, inpenetrable codebase. Internal peer review has made our code more robust: review of recent improvements to the coupled cluster Monte Carlo revealed a subtle bias when MPI parallelisation was used. We have also found the community aspect in development to be important and have some unexpected benefits. Recently several of us realised we were all struggling with a similar limitation in the code base and, as a result, embarked jointly on the (thankless) task of re-engineering some core data structures to provide additional flexibility. It is unlikely this work would have taken place if everyone was instead just focussing on their own research project in isolation (which discourages this kind of improvement/tidying/maintance that benefits everyone) but doing so will actually open up new possibilities for all of us.

In other instances, we have been less successful. One project on improving parallel scaling ended up running for almost a year, completely separate from the rest of the development. Combining this with other work was painful: such large sets of changes are hard to review adequately and the resultant merge had lots of conflicts which had to be resolved manually. We should have instead broken this work up into smaller sections rather than aiming for perfection in the first instance: our development model is better suited to continual refinement and incremental steps than large, radical changes. Another example is from legacy work: a seemingly innocuous (largely stylistic) change three years ago introduced a bug in an extreme corner case which, naturally, was eventually triggered. The problematic code dated back to before we systematically performed code reviews. The developer who found the bug was able to spot it quickly in the affected procedure, but tracking it down to that point from some unusual results in production calculations was *much* harder. The last two cases are not where our development approach failed *per se*, but rather where we failed it. Whilst there is always the temptation to follow the 'easy' course in the short term, in our experience this turns out to lead to pain later on—and often more quickly than anticipated!

As a project such as HANDE grows, there will be an increasing number of challenges in managing both the means of communication among the community as well as the direction of the project itself. To ensure community growth, it is vital that the low barrier of entry be maintained, and one way we are planning to ensure this is to include developer tutorials which provide a step-by-step introduction to both the code *and* our development practices. Requiring novitiates to work through these tutorials has the three-fold goal of indoctrination into the coding and development standards, learning the structures of the project, and keeping the tutorials up-to-date themselves. Often such tutorials are created on an *ad hoc* basis, but such practices are to be encouraged so as to sustain the accessibility to all. Indeed, the creation of tutorials aimed at users and developers would be a good introductory project when coupled with peer review.

We end with emphasising the benefits of an open source, collaborative approach, which we wholeheartedly endorse to the wider community. A code which is well written and easily understandable makes it easier to spot mistakes, which can then be fixed quickly and results produced with an open source implementation can be reproduced with no ambiguity. This enables scientists to spend more time pursing new ideas and less time resolving problems already solved by other groups, hence reducing the *collective* time to productive science.

**Acknowledgments**— JSS and WMCF acknowledge the Thomas Young Centre under Grant No. TYC-101. WAV is grateful to EPSRC for a studentship, FDM for an Imperial College PhD scholarship, NSB to Trinity College, Cambridge for an External Research Studentship, JJS to the Royal Commission for the Exhibition of 1851 for a Research Fellowship and AJWT to the Royal Society for a University Research Fellowship. We acknoledge the Imperial College High Performance Computing Service and ARCHER via a RAP award and via the Materials Chemistry Consortium (Grant No. EP/L000202).

[^1]: <email@example.com>, <firstname.lastname@example.com>

[^2]: <firstname.lastname@example.com>, <email@example.com>