one. This often ends in software collapse.

Let's have another look at the Python example shown above. I have already explained why the scientific environment expects its result to be 2.5. But what's the perspective of the technological environment? It sees the code as a Python program, with no interpretation of names. The code runs without raising an exception, so there's no reason to suppose a bug. And the output is... 4. At least for Python versions before 3.0. With Python 3, the new semantics for float division change the output to 4.0. In fact, what the technological environment's perspective really says is that the output depends on the computational environment, even in the hypothetical complete absence of bugs. Most of today's scientific environments still have a hard time understanding and accepting this fact, which is why computational reproducibility is still an unsolved problem in practice, even though it is fully understood in theory.
Trustworthy software

Scientists have to trust their tools in order to do their work. Software is an increasingly important tool in scientific research. And it seems that scientists do trust their software. There are no outcries of despair along the lines of “my research software crashes all the time” or “my simulation software had five bug fixes this year, I wonder if any of these bugs affected my last paper”. It thus seems that scientific software is in much better shape than, say, our word processors or Web browsers. *David Soergel disagrees*, judging that “computational results are particularly prone to misplaced trust”. *Harold Thimbleby disagrees as well*, comparing today’s use of software in science to the use of statistics before clear standards for reliable practices were developed.

The problem with computational results is that, if they look plausible, you are looking at numbers that are neither obviously wrong nor obviously correct. It isn't obvious either what “correct” actually means. And there is no obvious strategy to reduce this uncertainty. It is fundamentally due to the complexity of the computations that are performed. There are of course various checks we can do. Run the software on simple problems for which we know the correct results, for example. But correctness doesn't generalize from simple to complex problems, in particular not for computation, *which is chaotic*: any change to the code or to the input data can change the result of a computation without any predictable bound.

Perhaps the best check we can do is solve the same problem with different software packages and compare the results. It is unlikely that different developers make the same mistake, so if we get the same output from multiple sources, it is much more trustworthy. Unfortunately, this approach is costly, and therefore rarely applied.

The way we deal with this fundamental uncertainty is unconditional optimism:
we trust results as long as they are not obviously wrong. Probably there are
also scientists who go for unconditional pessimism, and stop using computers.
I met a few researchers with this attitude in the 1990s, but I doubt that
anyone with this point of view could survive in today's research environments.

This problem isn't as specific to software as it may seem from my description
so far. Scientists have published mistaken results well before they had
computers. One technique to detect and then eliminate them is independent
critical examination: ask another expert on the topic, who was not involved
in the original work, to go through the paper and check everything checkable.
That was the original idea of peer review, introduced in the 1950s, when
specialization became so pronounced that journal editors were no longer
competent to judge themselves all the submissions they received. Peer review
no longer works very well today, for various reasons, but that's a different
story.

For software, we have never attempted independent critical examination,
with rare exceptions that are limited to small pieces of code. This is not only
due to a lack of resources and incentives. Software is much more difficult
to review than papers. The complete software stack behind any published
result is enormous, and just listing all its components is a non-trivial task.
Moreover, software packages change all the time, and since computation is
chaotic, review must be in principle be re-done after every change.

But all software is not equal. Some software is more reviewable than others.
I have identified five dimensions along which scientific software packages can
vary, making them more or less reviewable. These five dimensions are:

1. Wide-spectrum vs. situated software. Wide-spectrum software packages a lot of functionality, trying to satisfy a large number of applications. This makes it hard to review, but it also makes reviewing a worthy investment. Situated software is written for a narrow application scenario, making it simpler and thus easier to review.

2. Mature vs. experimental software. Mature software is stable, and thus easier to review than experimental software, which is a moving target.

3. Convivial vs. proprietary software. Convivial software is written with the goal of appropriation by users. They take the code and adapt it to their needs, rather than re-using somebody else's code as a black-box tool. Code that is easy to appropriate is also easier to review. Proprietary software, whose source code is not available, is very hard to review. In between, we have Open Source software, whose source
code can be inspected but was not specifically written for inspectability
by its users.

4. Transparent vs. opaque software. Transparent software produces results that are easy to check for correctness because of the simplicity of the operations. It is therefore much easier to review. Unfortunately, most scientific software is opaque, making it hard to check its output.

5. Few dependencies vs. many dependencies. All software depends on other software for its execution. At the minimum, an operating system plus a compiler or interpreter. Dependencies must be examined as well as part of a software review, as they can be more or less trustworthy. Obviously, this becomes a Herculean task for software that has thousands of dependencies.

For a more detailed discussion, and for suggestions on improving the situation,
see my preprint "Establishing trust in automated reasoning".
Turing machine

This page is *empty*
# Web of science

The web of observations, models, hypotheses, the tools used to obtain or refine them, plus a large collection of narratives addressed at various publics (students, decision makers, etc.). The knowledge graph (but in a wider sense than the term is used in information technology) that is constantly updated by scientific research.

Not to be confused with its *capitalized namesake*, which is a commercial database providing a restricted and projected (but very useful) view of this graph.
Web3

This page is empty
Webstrates

see https://webstrates.net/
Wiki

Recommended reading:

* Wiki as a pattern language, by Ward Cunningham and Michael W. Mehaffy