# Introduction to newLISP
## Welcome to newLISP

Welcome to this introduction to newLISP! You'll find newLISP easy to learn and powerful, combining some of the power and elegance of classic LISP with the facilities of a modern scripting language, such as regular expressions, network functions, Unicode support, multitasking, and many others.
This book is a straightforward and simple description of the basics of the language. You should be familiar with using a text editor, and ideally you'll have done a bit of scripting before, but previous programming experience or knowledge of LISP isn't required. I hope it's enough to get you started, and ready to start exploring the real power of newLISP.
I'm writing this on a MacOS X system, but it shouldn't make any difference if you're using Linux, Windows, or one of the many other platforms that newLISP supports. The examples are designed to be run in newLISP version 10.
This is an unofficial document - for the official and definitive description of newLISP refer to the excellent reference manual that is installed with the software.

## Resources

* the main newLISP web site, newlisp.org, which provides a helpful forum, code examples and documentation, and the latest newLISP software
* the newLISP on Noodles wiki, newlisp-on-noodles.org
* John Small's excellent 21 minute introduction, newLISP in 21 minutes, at newLISP in 21 minutes
* the fine newLISP dragonfly logo, fashioned from 11 pairs of parentheses, is designed by Brian Grayless (fudnik.com)
Thanks to everyone who has contributed to earlier versions of this document, by suggesting additions or finding errors. Keep them coming.

# The basics
## Downloading and installation

Installation instructions can be found at newlisp.org.

## Getting started

Once you've installed newLISP, there are various ways to run it. See the newLISP documentation for full details. The most direct way is to run the newLISP interpreter from the command line - in a console or terminal window - by typing the newlisp command.
This is good for trying out short expressions, testing ideas, and for debugging. You can write multi-line code in this environment by enclosing the lines between [cmd] and [/cmd]. In new versions you can simply use the enter key on a blank line to begin and also to end the multi-line block.
The visual interface to newLISP, newLISP-GS, provides a graphical toolkit for newLISP applications, and it also gives you a development environment in which to write and test code: the newLISP editor. On Windows, this is installed as a desktop icon and a folder in the Program Start menu. On MacOS X, an application package and icon is installed in the Applications folder. The newLISP-GS editor provides you with multiple tabbed windows, syntax colouring, and a monitor area for seeing the results of running your code.
You can also run the newLISP-GS editor from the command line: you can find the file at C:/Program Files/newlisp/newlisp-edit (Windows), or at /usr/bin/newlisp-edit (on Unix). (This is a newLISP source file, so you can look at the code too.)
You can edit newLISP scripts in your favourite text editor. On MacOS X you can use BBEdit, TextWrangler, or TextMate to run newLISP scripts, or you can use one of the pre-installed Unix text editors such as vim or emacs. On Windows you can use UltraEdit, EditPlus, or NotePad++, to name just a few. If you use Linux, you know more about text editors than I do, and you probably already have a preference.
The newLISP web site hosts configuration files for a number of popular editors at http://newlisp.org/index.cgi?Code_Contributions.
On Unix, the first line of a newLISP script should be:
or:
If you prefer to run newLISP in an external text editor, you have to use more println functions to see the value returned by every function or expression.
Generally, you end a newLISP script or a console session using the exit function:

## The three basic rules of newLISP

You have to learn just three basic rules to program in newLISP. Here's the first one:

## Rule 1: a list is a sequence of elements

A list is a sequence of elements enclosed in parentheses:
The list is the basic data structure in newLISP, and it's also the way you write your program code. But don't type these examples in just yet - there are two more rules to learn!

## Rule 2: the first element in a list is special

When newLISP sees a list, it treats the first element as a function, and then tries to use the remaining elements as the information the function needs.
This is a list of three elements: the function called +, followed by two numbers. When newLISP sees this list, it evaluates it and returns the value 4 (of course). Notice that the first element was treated by newLISP as a function, whereas the rest of the elements were interpreted as arguments to that function - numbers that the function expects.
Here are some more examples that demonstrate these first two rules:
returns 45. The + function adds up all the numbers in the list.
returns 12.3, the biggest number in the list. Again, there's no (reasonable) limit to the length of the list: if a function is happy to accept 137 items (which both max and + are), than you can pass it 137 items.
prints the string of characters the sun has put his hat on. (It also returns the string, which is why, when you're working in the console, you sometimes see things repeated twice.) The print function can print out a single string of characters, or you can supply a sequence of elements to print:
which prints the two numbers and the three strings (although not very well formatted, because you haven't met the format function yet).
The directory function:
produces a listing of the specified directory, in this case the root directory, "/":
It lists the current directory if you don't specify one:
There's a read-file function that reads in the contents of a text file:
Here the function wants a single argument - the file name - and returns the contents of the file to you, in a string.
These are typical examples of the building blocks of newLISP code - a list containing a function call, followed perhaps by any extra information the function requires. There are over 380 newLISP functions, and you can refer to the excellent newLISP reference manual for details of all of them and how to use them.
You can try these examples. If you're using newLISP at a terminal, just type them in. If you're typing the lines into a text editor and running it as a script, you won't necessarily see the result of a function call unless you enclose the expression in the println function. For example, type:
to print the results of the read-file function.
Every newLISP expression returns a value. Even a println function returns a value. You could say that the printing action is actually just a side effect, and its main task is to return a value. You might notice that when you use println interactively in a console window, you see the return value twice: once when it's printed, and again when the value is returned to the calling function (the top-most level, in this case).
There's one more useful thing to look at before you meet the third rule.

### Nested lists

You've already spotted one list nesting inside another. Here's another example:
When newLISP sees this, it thinks as follows:
Hmm. Let's start with the first of those inner lists. I can do
easily. The value of that is 3. I can also do the second list
easily enough. That evaluates to 7.
So if I replace these two inner lists with these values, I get
which is really easy. I'll return the value 21 for this expression.
See those two right parentheses at the end of the first line, after the 4? Both are essential: the first one finishes the (+ 3 4 list, and the second one finishes the multiplication operation that started with (*. When you start writing more complicated code, you'll find that you are putting lists inside lists inside lists inside lists, and you might be ending some of the more complicated definitions with half a dozen right parentheses. A good editor will help you keep track of them.
But you don't have to worry about white space, line terminators, various punctuation marks, or compulsory indentation. And because all your data and your code are stored in the same way, in lists, you can mix them freely. More on that later.
Some people worry about the proliferation of parentheses when they first see LISP code. Others refer to them as nail clippings or say that LISP stands for Lots of Irritating Silly Parentheses. But I prefer to think of the parentheses as small handles that enclose a newLISP thought:
When you're editing newLISP code in a good editor, you can easily move or edit a thought by grabbing its handles, and easily select a thought with a Balance Parentheses command. You'll soon find the parentheses more useful than you first thought!

### Quoting prevents evaluation

You can now meet the third rule of programming with newLISP:

## Rule 3: Quoting prevents evaluating

To stop newLISP evaluating something, quote it.
Compare these two lines:
The first line is a list which contains a function and two numbers. In the second line, the list is quoted - preceded by a single quote or apostrophe ('). You don't need to put another quote at the end, after the closing parenthesis, because one is sufficient.
For the first expression, newLISP does its job as usual, and enthusiastically evaluates the list, returning the number 4. But for the second expression, as soon as it sees the quotation mark, newLISP doesn't even think about evaluating the list by adding the numbers; it just returns the list, unevaluated.
This quotation mark does the same job in newLISP that opening and closing quotation marks do in written English - they inform the reader that the word or phrase is not to be interpreted normally, but treated specially in some way: a non-standard or ironic meaning, perhaps, something spoken by another person, or something not to be taken literally.
So why do you want to stop newLISP evaluating things? You'll soon meet some examples where you quote things to prevent newLISP thinking that the first item in a list is a function. For example, when you store information in a list, you don't want newLISP to evaluate them in the usual way:
You don't want newLISP to look for functions called 2006 or "Arthur". Besides, 2006 isn't a valid function name, because it starts with a digit, and function names can't start with a double quotation mark, so in either case your program will stop with an error. Therefore you quote the lists to stop their first elements being used as functions rather than data:
newLISP's ability to treat expressions as data - and data as expressions - is discussed in more detail later.
Use the vertical apostrophe (ASCII code 39) to quote lists and symbols. Sometimes, text editors or other programs change these simple vertical apostrophes into curly quotation marks. These don't do the same job, so you will have to change any smart quotes to vertical apostrophes.

### Symbols and quotes

A symbol is a newLISP thing with a name. You define something in your code and assign a name to it. Then you can refer to that something later on, using the name rather than the contents. For example, after typing this:
there's now a new symbol called alphabet whose value is a string consisting of the 26 letters of the alphabet. The set function stores the string of characters from a to z in the symbol alphabet. Now this symbol can be used elsewhere, and will evaluate to the alphabet whenever it's used. Whenever you want to use the 26 letters of the alphabet, you use this symbol without quoting it. For example, here's the upper-case function:
I use the symbol without quoting it, because I want newLISP to use the value of the symbol, not its name. I'm really not interested in upper-casing the word alphabet, but the alphabet itself. newLISP hasn't permanently changed the value of the symbol, in this example, because upper-case always creates and returns a new string, leaving the one stored in the symbol unchanged.
Symbols correspond to variables in other programming languages. In fact, newLISP doesn't use symbols quite as much as other languages use variables. This is partly because values are continually being returned by expressions and fed directly into other expressions without being stored. For example, in the following code each function hands its result directly to the next, enclosing function:
upper-case gives its return value directly to first, which gives its return value directly to println, which both prints it and gives you the string it printed as the return value. So there's less need to store values temporarily. But there are plenty of other places where you do want symbols.
Here are two more examples of symbol-quoting:
In the first example, I don't quote the (+ 2 2) list - newLISP evaluates this to 4 and then assigns 4 to the symbol x, which evaluates to 4:
In the second example I quote the list. This means that the symbol y is now holding a list rather than a number. Whenever newLISP sees the symbol y, it will return the list, rather than 4. (Unless, of course, you quote y first as well!)
By the way, throughout this document:
and output printed by the newLISP interpreter is usually shown

### Setting and defining symbols

There are various ways to create and set the value of symbols. You can use define or set, as follows:
set expects to be followed by a symbol, but evaluates its first argument first. So you should either quote a symbol to prevent it being evaluated (because it might evaluate to something other than a symbol), or supply an expression that evaluates to a symbol. define doesn't expect the argument to be quoted.
You can also use setf and setq to set the value of symbols. These expect a symbol or a symbol reference as the first argument, so you don't have to quote it.
These two functions (which have the same action) can set the contents of a symbol (variable), a list, an array, or a string. A convention is to use setq when setting a symbol, and setf when setting an element of a list or array.
define is also used to define functions. See Make your own functions.

## Destructive functions

Some newLISP functions modify the values of the symbols that they operate on, others create a copy of the value and return that. Technically, the ones that modify the contents of symbols are described as destructive functions - although you'll often be using them to create new data. In this document I'll describe functions such as push and replace as destructive. This simply means that they change the value of something rather than return a modified copy.

# Controlling the flow

There are many different ways to control the flow of your code. If you've used other scripting languages, you'll probably find your favourites here, and many more besides.
All the control flow functions obey the standard rules of newLISP. The general form of each is usually a list in which the first element is a keyword followed by one or more expressions to be evaluated:

## Tests: if...

Perhaps the simplest control structure you can write in any language is a simple if list, consisting of a test and an action:
The second expression, a call to the launch-missile function, is evaluated only if the symbol button-pressed? evaluates to true. 1 is true. 0 is true - it's a number, after all. -1 is true. Most of the things that newLISP already knows about are true. There are two important things that newLISP knows are false rather than true: nil and the empty list. And anything that newLISP doesn't know the value of is false.
You can use anything that evaluates to either true or false as a test:
If a symbol evaluates to nil (perhaps because it doesn't exist or hasn't been assigned a value), newLISP considers it false and the test returns nil (because there was no alternative action provided):
You can add a third expression, which is the else action. If the test expression evaluates to nil or, the third expression is evaluated, rather than the second, which is ignored:
Here's a typical real-world three-part if function, formatted to show the structure as clearly as possible:
Although there are two expressions after the test - (net-flush) and (finish ...) - only one of them will be evaluated.
The lack of the familiar signpost words such as then and else that you find in other languages, can catch you out if you're not concentrating! But you can easily put comments in.
You can use if with an unlimited number of tests and actions. In this case, the if list consists of a series of test-action pairs. newLISP works through the pairs until one of the tests succeeds, then evaluates that test's corresponding action. If you can, format the list in columns to make the structure more apparent:
If you've used other LISP dialects, you might recognise that this is a simple alternative to cond, the conditional function. newLISP provides the traditional cond structure as well. See Selection: if, cond, and case.
You might be wondering how to do two or more actions if a test is successful or not. There are two ways to do this. You can use when, which is like an if without an 'else' part.
Another way is to define a block of expressions that form a single expression that you can use in an if expression. I'll discuss how to do this shortly, in Blocks: groups of expressions.
Earlier, I said that when newLISP sees a list, it treats the first element as a function. I should also mention that it evaluates the first element first before applying it to the arguments:
Here, the first element of the expression, (if (< x 5) + *), returns an arithmetic operator depending on the results of a test comparing x with 5. So the whole expression is either an addition or multiplication, depending on the value of x.
This technique can help you write concise code. Instead of this:
you could write this:
which evaluates like this:
Notice how each expression returns a value to the enclosing function. In newLISP every expression returns some value, even an if expression:
The value of x depends on the value returned by if expression. Now the symbol result contains a string depending on the value of x.

## Looping

Sometimes you want to repeat a series of actions more than once, going round in a loop. There are various possibilities. You might want to do the actions:
* on every item in a list
* on every item in a string
* a certain number of times
* until something happens
* while some condition prevails
newLISP has a solution for all of these, and more.

### Working through a list

newLISP programmers love lists, so dolist is a most useful function that sets a local loop symbol (variable) to each item of a list in turn, and runs a series of actions on each. Put the name for the loop variable and the list to be scanned, in parentheses, after dolist, then follow it with the actions.
In the following example, I set another symbol called counter as well, before defining a local loop variable i that will hold each value of the list of numbers generated by the sequence function:
Notice that, unlike if, the dolist function and many other control words let you write a series of expressions one after the other: here both the println and the inc (increment) functions are called for each element of the list.
There's a useful shortcut for accessing a system-maintained loop counter. Just now I used a counter symbol, incremented each time through the loop, to keep track of how far we'd got in the list. However, newLISP automatically maintains a loop counter for you, in a system variable called $idx, so I can dispense with the counter symbol, and just retrieve the value of $idx each time through the loop:
In some situations, you might prefer to use the mapping function map for processing a list (described later - see Apply and map: applying functions to lists). map can be used to apply a function (either an existing function or a temporary definition) to every element in a list, without going through the list using a local variable. For example, let's use map to produce the same output as the above dolist function. I define a temporary print and increase function consisting of two expressions, and apply this to each element of the list produced by sequence:
Experienced LISP programmers may be more familiar with lambda. fn is a synonym for lambda: use whichever one you prefer.
You might also find flat useful for working through lists, because it flattens out lists containing nested lists for easier processing, by copying:
to
for example. See flat.
To work through the arguments supplied to a function, you can use the doargs function. See Arguments: args.

### Working through a string

You can step through every character in a string using the equivalent of dolist, dostring.
The numbers are the ASCII/Unicode codes.

### A certain number of times

If you want to do something a fixed number of times, use dotimes or for. dotimes carries out a given number of repeats of the actions in the body of the list. You should provide a name for the local variable, just like you did with dolist:
You must supply a local variable with these forms. Even if you don't use it, you have to provide one.
Notice that counting starts at 0 and continues to n - 1, never actually reaching the specified value. Programmers think this is sensible and logical; non-programmers just have to get used to starting their counting at 0 and specifying 10 to get 0 to 9.
One way to remember this is to think about birthdays. You celebrate your 1st birthday when you complete your first year, during which time you were 0 years old. You complete your first 10 years of life when you start celebrating your 10th birthday, which starts when you stop being 9. The newLISP function first gets the element with index number 0...
Use dotimes when you know the number of repetitions, but use for when you want newLISP to work out how many repetitions should be made, given start, end, and step values:
Here newLISP is smart enough to work out that I wanted to step down from 1 to -1 in steps of 0.5.
Just to remind you of this counting from 0 thing again, compare the following two functions:

### An escape route is available

for, dotimes, and dolist like to loop, carrying out a set of actions over and over again. Usually the repetition continues, inexorably, until the limit - the final number, or the last item in the list - is reached. But you can specify an emergency escape route in the form of a test to be carried out before the next loop starts. If this test returns true, the next loop isn't started, and the expression finishes earlier than usual. This provides you with a way of stopping before the official final iteration.
For example, suppose that you want to halve every number in a list, but (for some reason) wanted to stop if one of the numbers was odd. Compare the first and second versions of this dolist expression:
The second version stops looping if the test for n being odd, (!= (mod n 2) 0), returns true.
Notice the use of integer-only division here. I've used / rather than the floating-point division operator div as part of the example. Don't use one if you want the other!
You can supply escape route tests with for and dotimes too.
For more complex flow control, you can use catch and throw. throw passes an expression to the previous catch expression which completes and returns the value of the expression:
The output is:
and the catch expression returns i was 5.
You can also devise flows using Boolean functions. See Blocks: groups of expressions.

### Until something happens, or while something is true

You might have a test for a situation that returns nil or when something interesting happens, but otherwise returns a true value, which you're not interested in. To repeatedly carry out a series of actions until the test fails, use until or do-until:
The difference between these two is to do with when the test is carried out. In until, the test is made first, then the actions in the body are evaluated if the test fails. In do-until, the actions in the body are evaluated first, before the test is made, then the test is made to see if another loop is possible.
Which of those two fragments of code is correct? Well, the first one tests the capacity of the disk before adding a file, but the second one, using do-until, doesn't check for free disk space until the file is added, which isn't so cautious.
while and do-while are the complementary opposites of until and do-until, repeating a block while a test expression remains true.
Choose the do- variants of each to do the action block before evaluating the test.

## Blocks: groups of expressions

A lot of newLISP control functions let you construct a block of actions: a group of expressions that are evaluated one by one, one after the other. Construction is implicit: you don't have to do anything except write them in the right order and in the right place. Look at the while and until examples above: each has three expressions that will be evaluated one after the other.
However, you can also create blocks of expressions explicitly using the begin, or, and and functions.
begin is useful when you want to explicitly group expressions together in a single list. Each expression is evaluated in turn:
and so on. You normally use begin only when newLISP is expecting a single expression. You don't need it with dotimes or dolist constructions, because these already allow more than one expression.
It doesn't matter what the result of each expression in a block is, unless it's bad enough to stop the program altogether. Returning nil is OK:
The values returned by each expression in a block are, in the case of begin, thrown away; only the value of the last expression is returned, as the value of the entire block. But for two other block functions, and and or, the return values are important and useful.

### and and or

The and function works through a block of expressions but finishes the block immediately if one of them returns nil (false). To get to the end of the and block, every single expression has to return a true value. If one expression fails, evaluation of the block stops, and newLISP ignores the remaining expressions, returning nil so that you know it didn't complete normally.
Here's an example of and that tests whether disk-item contains a useful directory:
The disk-item has to pass all three tests: it must be a directory, it mustn't be the . directory, and it mustn't be the .. directory (Unix terminology). When it successfully gets past these three tests, evaluation continues, and the message was printed. If one of the tests failed, the block completes without printing a message.
You can use and for numeric expressions too:
which tests whether c is between 33 and 255 inclusive and not equal to 48. This will always return either true or nil, depending on the value of c.
In some circumstances and can produce neater code than if. You can rewrite this example on the previous page:
to use and instead:
You could also use when here:
The or function is more easily pleased than its counterpart and. The series of expressions are evaluated one at a time until one returns a true value. The rest are then ignored. You could use this to work through a list of important conditions, where any one failure is important enough to abandon the whole enterprise. Or, conversely, use or to work through a list where any one success is reason enough to continue. Whatever, remember that as soon as newLISP gets a non-nil result, the or function completes.
The following code sets up a series of conditions that each number must avoid fulfilling - just one true answer and it doesn't get printed:

## ambiguity: the amb function

You may or may not find a good use for amb - the ambiguous function. Given a series of expressions in a list, amb will choose and evaluate just one of them, but you don't know in advance which one:
Use it to choose alternative actions at random:

## Selection: if, cond, and case

To test for a series of alternative values, you can use if, cond, or case. The case function lets you execute expressions based on the value of a switching expression. It consists of a series of value/expression pairs:
newLISP works through the pairs in turn, seeing if n matches any of the values 1, 2, 3, or 4. As soon as a value matches, the expression is evaluated and the case function finishes, returning the value of the expression. It's a good idea to put a final pair with true and a catch-all expression to cope with no matches at all. n will always be true, if it's a number, so put this at the end.
The potential match values are not evaluated. This means that you can't write this:
even though this ought logically to work: if n is 1, you would expect the first expression (- 2 1) to match. But that expression hasn't been evaluated - none of the sums have. In this example, the true action (println "je ne sais quoi") is evaluated.
If you prefer to have a version of case that evaluates its arguments, that's easily done in newLISP. See Macros.
Earlier I mentioned that cond is a more traditional version of if. A cond statement in newLISP has the following structure:
where each list consists of a test followed by one or more expressions or actions that are evaluated if the test returns true. newLISP does the first test, does the actions if the test is true, then ignores the remaining test/action loops. Often the test is a list or list expression, but it can also be a symbol or a value.
A typical example looks like this:
which is essentially the same as the if version, except that each pair of test-actions is enclosed in parentheses. Here's the if version for comparison:
For simpler functions, it might be easier to work with if. But cond can be more readable when writing longer programs. If you want the action for a particular test to evaluate more than one expression, cond's extra set of parentheses can give you shorter code:

## Variables local to a control structure

The control functions dolist, dotimes, and for involve the definition of temporary local symbols, which survive for the duration of the expression, then disappear.
Similarly, with the let and letn functions, you can define variables that exist only inside a list. They aren't valid outside the list, and they lose their value once the list has finished being evaluated.
The first item in a let list is a sublist containing variables (which don't have to be quoted) and expressions to initialize each variable. The remaining items in the list are expressions that can access those variables. It's a good idea to line up the variable/starting value pairs:
This example creates three local variables, x, y, and z, and assigns values to each. The body contains three println expressions. After these finish, the values of x, y, and z are no longer accessible - although the entire expression returns the value 16, the value returned by the final println statement.
The structure of a let list is easily remembered if you imagine it on a single line:
If you want to refer to a local variable elsewhere in the first, initialization, section, use letn rather than let:
In the definition of y, you can refer to the value of x, which we've only just defined to be 2. letn, a nested version of let, allows you to do this.
Our discussion of local variables leads to functions.

## Make your own functions

The define function provides a way to store a list of expressions under a name, suitable for running later. The functions you define can be used in the same way as newLISP's built-in functions. The basic structure of a function definition is like this:
when you don't want to supply any information to the function, or like this, when you do:
You call your newly defined function like any other function, passing values to it inside the list if your definition requires them:
I say expected, but newLISP is flexible. You can supply any number of arguments to func1, and newLISP won't complain. You can also call func2 with any number of arguments - in which case a, b, and c are set to nil at the start if there aren't enough arguments to define them.
When the function runs, each expression in the body is evaluated in sequence. The value of the last expression to be evaluated is returned as the function's value. For example, this function returns either true or nil, depending on the value of n:
Sometimes you'll want to explicitly specify the value to return, by adding an expression at the end that evaluates to the right value:
The message at the end evaluates to the message received and returned by (record-message) and the (answerphone) function returns this value. Without this, the function would return the value returned by (put-down-phone), which might be just a true or false value.
To make a function return more than one value, you can return a list.
Symbols that are defined in the function's argument list are local to the function, even if they exist outside the function beforehand:
If a symbol is defined inside a function body like this:
it's accessible from outside the function as well. That's why you'll want to define local variables! See Local variables.
newLISP is smart enough not to worry if you supply more than the required information:
but it won't fill in gaps for you:

## Local variables

Sometimes you want functions that change the values of symbols elsewhere in your code, and sometimes you want functions that don't - or can't. The following function, when run, changes the value of the x symbol, which may or may not be defined elsewhere in your code:
If you don't want this to happen, use let or letn to define a local x, which doesn't affect the x symbol outside the function:
x is still 10 outside the function. The x inside the function is not the same as the x outside. When you use set to change the value of the local x inside the function, it doesn't change any x outside:
instead of let and letn you can use the local function. This is like let and letn, but you don't have to supply any values for the local variables when you first mention them. They're just nil until you set them:
There are other ways of declaring local variables. You might find the following technique easier to write when you're defining your own functions. Watch the comma:
The comma is a clever trick: it's an ordinary symbol name like c or x:
- it's just less likely that you'd use it as a symbol name, and so it's useful as a visual separator in argument lists.

### Default values

In a function definition, the local variables that you define in the function's argument list can have default values assigned to them, which will be used if you don't specify values when you call the function. For example, this is a function with three named arguments, a, b, and c:
The symbols a and c will take the values 1 and 2 if you don't supply values in the function call, but b will be nil unless you supply a value for it.
1 nil 2
2 nil 2
2 3 2
3 2 1

### Arguments: args

You can see that newLISP is very flexible with its approach to arguments to functions. You can write definitions that accept any number of arguments, giving you (or the caller of your functions) maximum flexibility.
The args function returns any unused arguments that were passed to a function:
Notice that v1 contains the first argument passed to the function, but that any remaining unused arguments are in the list returned by (args).
With args you can write functions that accept different types of input. Notice how the following function can be called without arguments, with a string argument, with numbers, or with a list:
args allows you to write functions that accept any number of arguments. For example, newLISP is perfectly happy for you to pass a million arguments to a suitably defined function. I tried it:
In practice, newLISP was happy with this but my text editor wasn't.
The doargs function can be used instead of dolist to work through the arguments returned by args. You could have written the flexible function as:
newLISP has yet more ways to control the flow of code execution. As well as catch and throw, which allow you to handle and trap errors and exceptions, there's silent, which operates like a quiet version of begin.
If you want more, you can write your own language keywords, using newLISP macros, which can be used in the same way that you use the built-in functions. See Macros.

### Scope

Consider this function:
Notice that this function refers to some unspecified symbol x. This may or may not exist when the function is defined or called, and it may or may not have a value. When this function is evaluated, newLISP looks for the nearest symbol called x, and finds its value:
You can see how newLISP always gives you the value of the current x by dynamically keeping track of which x is active, even though there might be other xs lurking in the background. As soon as the for loop starts, the loop variable x takes over as the current x, but then that x is immediately superseded by the list iteration variable x which takes the value of a few trigonometric functions. In between each set of trig functions, the loop variable version of x pops back again briefly. And after all that iteration, the string value is available again.
In the func function, there's another x which is local to the function. When show is called, it will print this local symbol. A final call to show returns the very first value that x had.
Although newLISP won't get confused with all those different x's, you might! So it's a good idea to use longer and more explanatory symbol names, and use local rather than global variables. If you do, there's less chance of you making mistakes or of misreading your code at a later date. In general it's not a good idea to refer to an undefined symbol in a function unless you know exactly where it came from and how its value is determined.
This dynamic process of keeping track of the current version of a symbol is called dynamic scoping. There'll be more about this topic when you look at contexts (Contexts). These offer an alternative way to organize similarly-named symbols - lexical scoping.

# Lists

Lists are used everywhere in newLISP - LISP stands for list processing - so it's not surprising that there are many useful functions for working with lists. It's quite hard to organize them all into one logical descriptive narrative, but here's an introduction to most of them.
A good thing about newLISP is that many of the functions that work on lists also work on strings, so you'll meet many of these again in the next chapter, where they'll be applied to strings.

## Building lists

You can build a list and assign it to a symbol directly. Quote the list to stop it being evaluated immediately:
Lists are often created for you by other functions, but you can build your own too, using one of these functions:
* list makes a new list from expressions
* append glues lists together to form a new list
* cons prepends an element to a list or makes a list
* push inserts a new member in a list
* dup duplicates an element

### list, append, and cons

Use list to build a list from a sequence of expressions:
Notice that the (3 4) element is itself a list, which is nested inside the main list.
cons accepts exactly two expressions, and can do two jobs: insert the first element at the start of an existing list, or build a new two element list. In both cases it returns a new list. newLISP automatically chooses which job to do depending on whether the second element is a list or not.
To glue two or more lists together, use append:
Notice the difference between list and append when you join two lists:
list preserves the source lists when making the new list, whereas append uses the elements of each source list to make a new list.
To remember this: List keeps the List-ness of the source lists, but aPPend Picks the elements out and Packs them all together again.
append can also assemble a bunch of strings into a new string.

### push: pushing items onto lists

push is a powerful command that you can use to make a new list or insert an element at any location in an existing list. Pushing an element at the front of the list moves everything one place to the right, whereas pushing an element at the end of the list just attaches it and makes a new last element. You can also insert an element anywhere in the middle of a list.
Despite its constructive nature, it's technically a destructive function, because it changes the target list permanently, so use it carefully. It returns the value of the element that was inserted, rather than the whole list.
When you refer to the location of an element in a list, you use zero-based numbering, which you would expect if you're an experienced programmer:
If you don't specify a location or index, push pushes the new element at the front. Use a third expression to specify the location or index for the new element. -1 means the last element of the list, 1 means the second element of the list counting from the front from 0, and so on:
If the symbol you supply as a list doesn't exist, push usefully creates it for you, so you don't have to declare it first.
By the way, there are plenty of other ways of generating a list of unsorted numbers. You could also do a number of random swaps like this:
although it would be even easier to use randomize:
(That's one of the nice things about newLISP - a more elegant solution is just a re-write away!)
push has an opposite, pop, which destructively removes an element from a list, returning the removed element. We'll meet pop and other list surgery functions later. See List surgery.
These two functions, like many other newLISP functions, work on strings as well as lists. See push and pop work on strings too.

### dup: building lists of duplicate elements

A useful function called dup lets you construct lists quickly by repeating elements a given number of times:
There's a trick to get dup to return a list of strings. Because dup can also be used to duplicate strings into a single longer string, you supply an extra true value at the end of the list, and newLISP creates a list of strings rather than a string of strings.

## Working with whole lists

Once you've got a list, you can start to work on it. First, let's look at the functions that operate on the list as a unit. After that, I'll look at the functions that let you carry out list surgery - operations on individual list elements.

### Using and processing lists

dolist works through every item of a list:
In this example, apply expects a function and a list, and uses the elements of that list as arguments to the function. So it repeatedly applies the upper-case function to the loop variable's value in v. Since upper-case works on strings but apply expects a list, I've had to use the list function to convert the current value of v (a string) in each iteration to a list.
A better way to do this is to use map:
map applies the named function, upper-case in this example, to each item of the list in turn. The advantage of map is that it both traverses the list and applies the function to each item of the list in one pass. The result is a list, too, which might be more useful for subsequent processing.
There's more about dolist and apply elsewhere (see Working through a list, and Apply and map: applying functions to lists).

### reverse

reverse does what you would expect, and reverses the list. It's a destructive function, changing the list permanently.

### sort and randomize

In a way, randomize and sort are complementary, although sort changes the original list, whereas randomize returns a disordered copy of the original list. sort arranges the elements in the list into ascending order, organizing them by type as well as by value.
Here's an example: create a list of letters and a list of numbers, stick them together, shuffle the result, and then sort it again:
Compare data before it was randomized and after it was sorted. The sort command sorts the list by data type as well as by value: integers before strings, strings before lists, and so on.
The default sorting method is <, which arranges thet values so that each is less than the next.
To change the sort method, you can supply one of newLISP's built-in comparison functions, such as >. Adjacent objects are considered to be correctly sorted when the comparison function is true for each adjacent pair:
You can supply a custom sort function. This is a function that takes two arguments and returns true if they're in the right order (ie if the first should come before the second), and false if they aren't. For example, suppose you want to sort a list of file names so that the shortest name appears first. Define a function that returns true if the first argument is shorter than the second, and then use sort with this custom sorting function:
Advanced newLISPers often write a nameless function and supply it directly to the sort command:
This does the same job, but saves 25 characters or so. You can use either fn or lambda to define an inline or anonymous function.

### unique

unique returns a copy of a list with all duplicates removed:
There are also some useful functions for comparing lists. See Working with two or more lists.

### flat

flat is useful for dealing with nested lists, because it shows what they look like without a complicated hierarchical structure:
Fortunately, flat is non-destructive, so you can use it without worrying about losing the structure of nested lists:

### transpose

transpose is designed to work with matrices (a special type of list: see Matrices). It also does something useful with ordinary nested lists. If you imagine a list of lists as a table, it flips rows and columns for you:
And here's a nice bit of newLISP sleight of hand:
and each sublist has been reversed. You could, of course, do this:
which is shorter, but a bit slower.

### explode

The explode function lets you blow up a list:
You can specify the size of the pieces, too:

## List analysis: testing and searching

Often you don't know what's in a list, and you want some forensic tools to find out more about what's inside it. newLISP has a good selection.
We've already met length, which finds the number of elements in a list.
The starts-with and ends-with functions test the start and ends of lists:
These functions work equally well with strings, too (and they take regular expressions). See Testing and comparing strings.
How about contains? In fact there isn't one single newLISP function that does this job. Instead you have find, match, member, ref, filter, index, and count, among others. Which one you use depends on what sort of answer you want to the question Does this list contain this item?, and whether the list is a nested list or a flat one.
If you want a simple answer with a quick top-level-only search, use find. See find.
If you want the item and the rest of the list as well, use member. See member.
If you want the index number of the first occurrence, even if the list contains nested lists, you can use ref. See ref and ref-all.
If you want a new list containing all the elements that match your search element, use find-all. See find-all.
If you want to know whether or not the list contains a certain pattern of elements, use match. See matching patterns in lists.
You can find all list items that satisfy a function, either built-in or one of your own, with the filter, clean, and index functions. See Filtering lists: filter, clean, and index.
The exists and for-all functions check elements in a list to see if they pass a test.
If you want to find elements in a list purely to change them to something else, then don't bother to look for them first, just use replace. See Replacing information: replace. You can also find and replace list elements using the set-ref function. See Find and replace matching elements.
If you want to know how many occurrences there are of items in the list, use count. See Working with two or more lists.
Let's look at some examples of each of these.

### find

find looks through a list for an expression and returns an integer or nil. The integer is the index of the first occurrence of the search term in the list. It's possible for find to return 0 - if the list starts with the item, its index number is 0, but this isn't a problem - you can use this function in an if test, because 0 evaluates to true.
Here I've parsed the text of Sir Arthur Conan Doyle's The Sign of Four (which you can download from Project Gutenberg), and tested whether the resulting list of strings contains various names. The integer returned is the first occurrence of the string element in the list.
find lets you use regular expressions, too, so you can find any string elements in a list that match a string pattern:
Here I'm looking for any traces of chemical indulgence in Sherlock Holmes' bohemian way of life: "(tea|cocaine|morphine|tobacco)" means any one of tea, cocaine, morphine, or tobacco.
This form of find lets you look for regular expression patterns in the string elements of a list. You'll meet regular expressions again when we explore strings. See Regular expressions.
Here we're looking for any string elements in the word-list that match our pattern (a c followed by ie, the old and inaccurate spelling rule i before e except after c).
The regular expression pattern here (enclosed in braces, which are string delimiters, doing more or less the same job as quotation marks) is (c) followed by (ie). Then there's a comment, starting with (?#. Comments in regular expressions are useful when things get cryptic, as they often do.
find can also accept a comparison function. See Searching lists.
find finds only the first occurrence in a list. To find all occurrences, you could keep repeating find until it returns nil. The word list gets shorter each time through, and the found elements are added at the end of another list:
But in this case it's much easier to use filter:
- see Filtering lists: filter, clean, and index.
Alternatively, you can use ref-all (see ref and ref-all) to get a list of indices.
If you're not using regular expressions, you can use count, which, given two lists, looks through the second list and counts how many times each item in the first list occurs. Let's see how many times the main characters' names are mentioned:
The list of results produced by count shows how many times each element in the first list occurs in the second list, so there are 34 mentions of Sherlock, 135 mentions of Holmes, 24 of Watson, and only one mention for poor old Inspector Lestrade in this story.
It's worth knowing that find examines the list only superficially. For example, if the list contains nested lists, you should use ref rather than find, because ref looks inside sublists:

### member

The member function returns the rest of the source list rather than index numbers or counts:

### matching patterns in lists

There's a powerful and complicated function called match, which looks for patterns in lists. It accepts the wild card characters *, ?, and +, which you use to define a pattern of elements. + means one or more elements, * means zero or more elements, and ? means one element. For example, suppose you want to examine a list of random digits between 0 and 9 for patterns. First, generate a list of 10000 random numbers as source data:
Next, you decide that you want to find the following pattern:
somewhere in the list, ie anything followed by 1 followed by a 2 followed by a 3 followed by anything. Call match like this:
which looks odd, but it's just a pattern specification in a list, followed by the source data. The list pattern:
means any sequence of atoms or expressions (or nothing), followed by a 1, then a 2, then a 3, followed by any number of atoms or expressions or nothing. The answer returned by this match function is another list, consisting of two sublists, one corresponding to the first *, the other corresponding to the second:
and the pattern you were looking for first occurred in the gap between these lists (in fact it occurs half a dozen times later on in the list as well). match can also handle nested lists.
To find all occurrences of a pattern, not just the first, you can use match in a while loop. For example, to find and remove every 0 when it's followed by another 0, repeat the match on a new version of the list until it stops returning a non-nil value:
You don't have to find elements first before replacing them: just use replace, which does the finding and replacing in one operation. And you can use match as a comparison function for searching lists. See Replacing information: replace, and Searching lists.

### find-all

find-all is a powerful function with a number of different forms, suitable for searching lists, association lists, and strings. For list searches, you supply four arguments: the search key, the list, an action expression, and the functor, which is the comparison function you want to use for matching the search key:
Here, find-all is looking for the string "onion" in the list food. It's using the > function as a comparison function, so it will find anything that "onion" is greater than. For strings, 'greater than' means appearing later in the default ASCII sorting order, so that "cheese" is greater than "bread" but less than "onion". Notice that, unlike other functions that let you provide comparison functions (namely find, ref, ref-all, replace when used with lists, set-ref, set-ref-all, and sort), the comparison function must be supplied. With the < function, the result is a list of things that "onion" is less than:

### ref and ref-all

The ref function returns the index of the first occurrence of an element in a list. It's particularly suited for use with nested lists, because, unlike find, it looks inside all the sublists, and returns the address of the first appearance of an element. As an example, suppose you've converted an XML file, such as your iTunes library, into a large nested list, using newLISP's built-in XML parser:
Now you can look for any expression in the data, which is in the form of an ordinary newLISP list:
and the returned list will be the location of the first occurrence of that string in the list:
- this is a set of index numbers which together define a kind of address. This example means: in list element 0, look for sublist element 2, then find sublist element 14 of that sublist, and so on, drilling down into the highly-nested XML-based data structure. See Working with XML.
ref-all does a similar job, and returns a list of addresses:
These functions can also accept a comparison function. See Searching lists.
Use these functions when you're searching for something in a nested list. If you want to replace it when you've found it, use the set-ref and set-ref-all functions. See Find and replace matching elements.

### Filtering lists: filter, clean, and index

Another way of finding things in lists is to filter the list. Like panning for gold, you can create a filter that keeps only the stuff you want, flushing the unwanted stuff away.
The functions filter and index have the same syntax, but filter returns the list elements, whereas index returns the index numbers (indices) of the wanted elements rather than the list elements themselves. (These functions don't work on nested lists.)
The filtering functions filter, clean, and index use another function for testing elements: the element appears in the results list according to whether it passes the test or not. You can either use a built-in function or define your own. Typically, newLISP functions that tests and return true or false (sometimes called predicate functions) have names ending with question marks:
NaN? array? atom? context? directory? empty? file? float? global? integer? lambda? legal? list? macro? nil? null? number? primitive? protected? quote? string? symbol? true? zero?
So, for example, an easy way to find integers in (and remove floating-point numbers from) a list is to use the integer? function with filter. Only integers pass through this filter:
filter has a complementary function called clean which removes elements that satisfy the test:
Think of clean as getting rid of dirt - it gets rid of anything that passes the test. Think of filter as panning for gold, keeping what passes the test.
This next filter finds all words in Conan Doyle's story The Empty House that contain the letters pp. The filter is a lambda expression (a temporary function without a name) that returns nil if the element doesn't contain pp. The list is a list of string elements generated by parse, which breaks up a string into a list of smaller strings according to a pattern.
You can also use filter or clean for tidying up lists before using them - removing empty strings produced by a parse operation, for example.
When would you use index rather than filter or clean? Well, use index when you later want to access the list elements by index number rather than their values: we'll meet functions for selecting list items by index in the next section. For example, whereas ref found the index of only the first occurrence, you could use index to return the index numbers of every occurrence of an element.
If you have a predicate function that looks for a string in which the letter c is followed by ie, you can use that function to search a list of matching strings:
Remember that lists can contain nested lists, and that some functions won't look inside the sub-lists:

### Testing lists

The exists and for-all functions check elements in a list to see if they pass a test.
exists returns either the first element in the list that passes the test, or nil if none of them do.
for-all returns either true or nil. If every list element passes the test, it returns true.

### Searching lists

find, ref, ref-all and replace look for items in lists. Usually, you use these functions to find items that equal what you're looking for. However, equality is just the default test: all these functions can accept an optional comparison function that's used instead of a test for equality. This means that you can look for list elements that satisfy any test.
The following example uses the < comparison function. find looks for the first element that compares favourably with n, ie the first element that n is less than. With a value of 1002, the first element that satisfies the test is 1003, the 3rd element of the list, and so the returned value is 3.
You can write your own comparison function:
The longer? function returns true if the first argument is longer than the second. So find, with this function as the comparison, finds the first element in the list that makes the comparison true. Because tiger is longer than dog, the function returns 3, the index of dog in the list.
You could supply an anonymous (or lambda) function as part of the find function, rather than write a separate function:
If you want your code to be readable, you'll probably move longer or more complex comparators out into their own separate - and documented - functions.
You can also use comparison functions with ref, ref-all, and replace.
A comparison function can be any function that takes two values and returns true or false. For example, here's a function that returns true if y is greater than 6 and less than x. The search is therefore for an element of the data list that is both smaller than the searched-for number, which in this case is 15, and yet bigger than 6.

### Summary: compare and contrast

To summarize these contains functions, here they are in action:

## Selecting items from lists

There are various functions for getting at the information stored in a list:
* first gets the first element
* rest gets all but the first element
* last returns the last element
* nth gets the nth element
* select selects certain elements by index
* slice extracts a sublist
The first and rest functions are more sensible names for the traditional car and cdr LISP functions, which were based on the names of old computer hardware registers.

### Picking elements: nth, select, and slice

nth gets the nth element of a list:
nth can also look inside nested lists, because it accepts more than one index number:
If you want to pick a group of elements out of a list, you'll find select useful. You can use it in two different forms. The first form lets you supply a sequence of loose index numbers:
A positive number selects an element by counting forward from the beginning, and a negative number selects by counting backwards from the end:
You can also supply a list of index numbers to select. For example, you can use the rand function to generate a list of 20 random numbers between 0 and 8, and then use this list to select elements from phrase at random:
Notice the duplicates. If you had written this instead:
there would be no duplicates: (randomize phrase) shuffles elements without duplicating them.
slice lets you extract sections of a list. Supply it with the list, followed by one or two numbers. The first number is the start location. If you miss out the second number, the rest of the list is returned. The second number, if positive, is the number of elements to return.
If negative, the second number specifies an element at the other end of the slice counting backwards from the end of the list, -1 being the final element:
The cake knife reaches as far as - but doesn't include - the element you specify.

## Implicit addressing

newLISP provides a faster and more efficient way of selecting and slicing lists. Instead of using a function, you can use index numbers and lists together. This technique is called implicit addressing.

### Select elements using implicit addressing

As an alternative to using nth, put the list's symbol and an index number in a list, like this:
If you have a nested list, you can supply a sequence of index numbers that identify the list in the hierarchy:
where the '(2 1) first finds element 2, ("lion" 4), then finds element 1 (the second one) of that sublist.

### Selecting a slice using implicit addressing

You can also use implicit addressing to get a slice of a list. This time, put one or two numbers to define the slice, before the list's symbol, inside a list:
Earlier, we parsed the iTunes XML library:
Let's access the inside of the resulting XML structure using the implicit addressing techniques:
How to remember the difference between the two types of implicit addressing? sLice numbers go in the Lead, sElect numbers go at the End.

## List surgery
### Shortening lists

To shorten a list, by removing elements from the front or back, use chop or pop. chop makes a copy and works from the end, pop changes the original and works from the front.
chop returns a new list by cutting the end off the list:
An optional third argument for chop specifies how many elements to remove:
pop (the opposite of push) permanently removes the specified element from the list, and works with list indices rather than lengths:
You can also use replace to remove items from lists.

### Changing items in lists

You can easily change elements in lists, using the following functions:
* replace changes or removes elements
* swap swaps two elements
* setf sets the value of an element
* set-ref searches a nested list and changes an element
* set-ref-all searches for and changes every element in a nested list
These are destructive functions, just like push, pop, reverse, and sort, and change the original lists, so use them carefully.

### Changing the nth element

To set the nth element of a list (or array) to another value, use the versatile setf command:
Notice how the setf function returns the value that has just been set, 0, rather than the changed list.
This example uses the faster implicit addressing. You could of course use nth to create a reference to the nth element first:
setf must be used on a list or array or element stored in a symbol. You can't pass raw data to it:

### Using it

Sometimes when you use setf, you want to refer to the old value when setting the new value. To do this, use the system variable $it. During a setf expression, $it contains the old value. So to increase the value of the first element of a list by 1:
You can do this with strings too. Here's how to 'increment' the first letter of a string:

### Replacing information: replace

You can use replace to change or remove elements in lists. Specify the element to change and the list to search in, and also a replacement if there is one.
Every matching item is deleted.
replace returns the changed list:
The replacement can be a simple value, or any expression that returns a value.
replace updates a set of system variables $0, $1, $2, up to $15, and the special variable $it, with the matching data. For list replacements, only $0 and $it are used, and they hold the value of the found item, suitable for using in the replacement expression.
For more about system variables and their use with string replacements, see System variables.
If you don't supply a test function, = is used:
You can make replace find elements that pass a different test, other than equality. Supply the test function after the replacement value:
The test can be any function that compares two values and returns a true or false value. This can be amazingly powerful. Suppose you have a list of names and their scores:
How easy is it to add up the numbers for all those people whose scores included a 0? Well, with the help of the match function, this easy:
Here, for each matching element, the replacement expression builds a list from the name and the sum of the scores. match is employed as a comparator function - only matching list elements are selected for totalization, so Eric's scores weren't totalled since he didn't manage to score 0.
See Changing substrings for more information about using replace on strings.

### Modifying lists

There are even more powerful ways of modifying the elements in lists. Meet set-ref and set-ref-all.
You can locate and modify elements using these functions, which are designed to work well with nested lists. (See also Working with XML for some applications.)

## Find and replace matching elements

The set-ref function lets you modify the first matching element in a list:
To change that 200 to a 300, use set-ref like this:

## Find and replace all matching elements: set-ref-all

set-ref finds the first matching element in a nested list and changes it; set-ref-all can replace every matching element. Consider the following nested list that contains data on the planets:
How could you change every occurrence of that 'incline' symbol to be 'inclination'? It's easy using set-ref-all:
This returns the list with every 'incline changed to 'inclination.
As with replace, the default test for finding matching elements is equality. But you can supply a different comparison function. This is how you could examine the list of planets and change every entry where the moon's value is greater than 9 to say "lots" instead of the actual number.
The replacement expression compares the number of moons (the last item of the result which is stored in $0) and evaluates to "lots" if it's greater than 9. The search term is formulated using match-friendly wildcard syntax, to match the choice of comparison function.

### Swap

The swap function can exchange two elements of a list, or the values of two symbols. This changes the original list:
Usefully, swap can also swap the values of two symbols without you having to use an intermediate temporary variable.
This parallel assignment can make life easier sometimes, such as in this slightly unusual iterative version of a function to find Fibonacci numbers:

## Working with two or more lists

If you have two lists, you might want to ask questions such as How many items are in both lists?, Which items are in only one of the lists?, How often do the items in this list occur in another list?, and so on. Here are some useful functions for answering these questions:
* difference finds the set difference of two lists
* intersect finds the intersection of two lists
* count counts the number of times each element in one list occurs in a second list
For example, to see how many vowels there are in a sentence, put the known vowels in one list, and the sentence in another (first use explode to turn the sentence into a list of characters):
or the slightly shorter:
The result, (1 4 1 4 2), means that there are 1 a, 4 e's, 1 i, 4 o's, and 2 u's in the sentence.
difference and intersect are functions that will remind you of those Venn diagrams you did at school (if you went to a school that taught them). In newLISP lists can represent sets.
difference returns a list of those elements in the first list that are not in the second list. For example, you could compare two directories on your system to find files that are in one but not the other. You can use the directory function for this.
It's important which list you put first! There are five files or directories in directory d1 that aren't in directory d2, but there are no files or directories in d2 that aren't also in d1.
The intersect function finds the elements that are in both lists.
Both these functions can take an additional argument, which controls whether to keep or discard any duplicate items.
You could use the difference function to compare two revisions of a text file. Use parse (Parsing strings) to split the files into lines first:

## Association lists

There are various techniques available in newLISP for storing information. One very easy and effective technique is to use a list of sublists, where the first element of each sublist is a key. This structure is known as an association list, but you could also think of it as a dictionary, since you look up information in the list by first looking for the key element.
You can also implement a dictionary using newLISP's contexts. See Introducing contexts.
You can make an association list using basic list functions. For example, you can supply a hand-crafted quoted list:
Or you could use functions like list and push to build the association list:
It's a list of sublists, and each sublist has the same format. The first element of a sublist is the key. The key can be a string, a number, or a symbol. You can have any number of data elements after the key.
Here's an association list that contains some data about the planets in the solar system:
Each sublist starts with a string, the name of a planet, which is followed by data elements, numbers in this case. The planet name is the key. I've included some comments at the end, because I'm never going to remember that element 2 is the planet's mass, in Earth masses.
You could easily access this information using standard list-processing techniques, but newLISP offers some tailor-made functions that are designed to work specifically with these dictionary or association lists:
* assoc finds the first occurrence of the keyword and return the sublist
* lookup looks up the value of a keyword inside the sublist
Both assoc and lookup take the first element of the sublist, the key, and retrieve some data from the appropriate sublist. Here's assoc in action, returning the sublist:
And here's lookup, which goes the extra mile and gets data out of an element of one of the sublists for you, or the final element if you don't specify one:
This saves you having to use a combination of assoc and nth.
One problem that you might have when working with association lists with long sublists like this is that you can't remember what the index numbers represent. Here's one solution:
Here we've defined orbital-radius and au (astronomical unit) as constants, and you can use orbital-radius to refer to the right column of a sublist. This also makes the code easier to read. The constant function is like set, but the symbol you supply is protected against accidental change by another use of set. You can change the value of the symbol only by using the constant function again.
Having defined these constants, here's an expression that lists the different orbits of the planets, in kilometres:
When you want to manipulate floating-point numbers, use the floating-point arithmetic operators add, sub, mul, div rather than +, -, *, and /, which work with (and convert values to) integers.

### Replacing sublists in association lists

To change values stored in an association list, use the assoc function as before, to find the matching sublist, then use setf on that sublist to change the value to a new sublist.

### Adding new items to association lists

Association lists are ordinary lists, too, so you can use all the familiar newLISP techniques with them. Want to add a new 10th planet to our sol-sys list? Just use push:
and check that it was added OK with:
You can use sort to sort the association list. (Remember though that sort changes lists permanently.) Here's a list of planets sorted by mass. Since you don't want to sort them by name, you use a custom sort (see sort and randomize) to compare the mass (index 2) values of each pair:
You can also easily combine the data in the association list with other lists:
Here we've created a temporary inline function that map applies to each planet in sol-sys - lookup finds the planet name and retrieves the Unicode symbol for that planet from the unicode-symbols association list.
You can quickly remove an element from an association list with pop-assoc.
This removes the Pluto element from the list.
newLISP offers powerful data storage facilities in the form of contexts, which you can use for building dictionaries, hash tables, objects, and so on. You can use association lists to build dictionaries, and work with the contents of dictionaries using association list functions. See Introducing contexts.
You can also use a database engine - see Using a SQLite database.

### find-all and association lists

Another form of find-all lets you search an association list for a sublist that matches a pattern. You can specify the pattern with wildcard characters. For example, here's an association list:
To find all the sublists that end with 9, use the match pattern (? 9), where the question mark matches any single item:
(For more about match patterns - wild card searches for lists - see matching patterns in lists.)
You can also use this form with an additional action expression after the association list:
Here, the action expression uses $0 to refer to each matched element in turn.

# Strings

String-handling tools are an important part of a programming language. newLISP has many easy to use and powerful string handling tools, and you can easily add more tools to your toolbox if your particular needs aren't met.
Here's a guided tour of newLISP's string orchestra.

## Strings in newLISP code

You can write strings in three ways:
* enclosed in double quotes
* embraced by curly braces
* marked-up by markup codes
like this:
All three methods can handle strings of up to 2048 characters. For strings longer than 2048 characters, always use the [text] and [/text] tags to enclose the string.
Always use the first method, quotation marks, if you want escaped characters such as \n and \t, or code numbers (\046), to be processed.
The double quotation character must be escaped with backslashes, as must a backslash, if you want them to appear inside a string.
Use the second method, braces (or 'curly brackets'), for strings shorter than 2048 characters when you don't want any escaped characters to be processed:
This is a really useful way of writing strings, because you don't have to worry about putting backslashes before every quotation character, or backslashes before other backslashes. You can nest pairs of braces inside a braced string, but you can't have an unmatched brace. I like to use braces for strings, because they face the correct way (which plain dumb quotation marks don't) and because your text editor might be able to balance and match them.
The third method, using [text] and [/text] markup tags, is intended for longer text strings running over many lines, and is used automatically by newLISP when it outputs large amounts of text. Again, you don't have to worry about which characters you can and can't include - you can put anything you like in, with the obvious exception of [/text]. Escape characters such as \n or \046 aren't processed either.
If you want to know the length of a string, use length:
Strings of millions of characters can be handled easily by newLISP.
Rather than length, use utf8len to get the length of a Unicode string:

## Making strings

Many functions, such as the file-reading ones, return strings or lists of strings for you. But if you want to build a string from scratch, one way is to start with the char function. This converts the supplied number to the equivalent character string with that code number. It can also reverse the operation, converting the supplied character string to its equivalent code number.)
These last two examples are available when you're running the Unicode-capable version of newLISP. Since Unicode is hexadecimally inclined, you can give a hex number, starting with 0x, to char. To see the actual characters, use a printing command:
λ
The backslashed numbers are the result of the println function, presumably the multi-byte values of the Unicode glyph.
You can use char to build strings in other ways:
This uses char to find out the ASCII code numbers for a and z, and then uses sequence to generate a list of code numbers between the two. Then the char function is mapped onto every element of the list, so producing a list of strings. Finally, this list is converted to a single string by join.
join can also take a separator when building strings:
Similar to join is append, which works directly on strings:
but even more useful is string, which turns any collection of numbers, lists, and strings into a single string.
Notice that the first list wasn't evaluated (because it was quoted) but that the second list was evaluated to produce a list of numbers, and the resulting list - including the parentheses - was converted to a string.
The string function, combined with the various string markers such as braces and markup tags, is one way to include the values of variables inside strings:
You can also use format to combine strings and symbol values. See Formatting strings.
dup makes copies:
And date makes a date string:
or you can give it a number of seconds since 1970 to convert:
See Working with dates and times.

## String surgery

Now you've got your string, there are plenty of functions for operating on them. Some of these are destructive functions - they change the string permanently, possibly losing information for ever. Others are constructive, producing a new string and leaving the old one unharmed. See Destructive functions.
reverse is destructive:
Now t has changed for ever. However, the case-changing functions aren't destructive, producing new strings without harming the old ones:

## Substrings

If you know which part of a string you want to extract, use one of the following constructive functions:
You can also use this technique with lists. See Selecting items from lists.

### String slices

slice gives you a new slice of an existing string, counting either forwards from the cut (positive integers) or backwards from the end (negative integers), for the given number of characters or to the specified position:
There's a shortcut to do this, too. Put the required start and length before the string in a list:
If you don't want a continuous run of characters, but want to cherry-pick some of them for a new string, use select followed by a sequence of character index numbers:
which is good for finding secret coded messages buried in text.

### Changing the ends of strings

trim and chop are both constructive string-editing functions that work from the ends of the original strings inwards.
chop works from the end:
trim can remove characters from both ends:

### push and pop work on strings too

You've seen push and pop adding and removing items from lists. They work on strings too. Use push to add characters to a string, and pop to remove one character from a string. Strings are added to or removed from the start of the string, unless you specify an index.
pop always returns what was popped, but push returns the modified target of the action. It's useful when you want to break up a string and process the pieces as you go. For example, to print the newLISP version number, which is stored as a 4 or 5 digit integer, use something like this:
It's easier to work from the right-hand side of the string and use pop to extract the information and remove it in one operation.

## Modifying strings

There are two approaches to changing characters inside a string. Either use the index numbers of the characters, or specify the substring you want to find or change.

### Using index numbers in strings

To change characters by their index numbers, use setf, the general purpose function for changing strings, lists, and arrays:
You could also use nth with setf to specify the location:
Here's how to 'increment' the first (zeroth) letter of a string:
$it contains the value found by the first part of the setf expression, and its numeric value is incremented to form the second part.

### Changing substrings

If you don't want to - or can't - use index numbers or character positions, use replace, a powerful destructive function that does all kinds of useful operations on strings. Use it in the form:
So:
replace is destructive, but if you want to use replace or another destructive function constructively for its side effects, without modifying the original string, use the copy function:
The copy is modified by replace. The original string t is unaffected.

### Regular expressions

replace is one of a group of newLISP functions that accept regular expressions for defining patterns in text. For most of them, you add an extra number at the end of the expression which specifies options for the regular expression operation: 0 means basic matching, 1 means case-insensitive matching, and so on.
Sometimes I put comments inside regular expressions, so that I know what I was trying to do when I read the code some days later. Text between (?# and the following closing parenthesis is ignored.
If you're happy working with Perl-compatible Regular Expressions (PCRE), you'll be happy with replace and its regex-using cousins (find, regex, find-all, parse, starts-with, ends-with, directory, and search ). Full details are in the newLISP reference manual.
You have to steer your pattern through both the newLISP reader and the regular expression processor. Remember the difference between strings enclosed in quotes and strings enclosed in braces? Quotes allow the processing of escaped characters, whereas braces don't. Braces have some advantages: they face each other visually, they don't have smart and dumb versions to confuse you, your text editor might balance them for you, and they let you use the more commonly occurring quotation characters in strings without having to escape them all the time. But if you use quotes, you must double the backslashes, so that a single backslash survives intact as far as the regular expression processor:

### System variables: $0, $1 ...

replace updates a set of system variables $0, $1, $2, up to $15, with the matches. These refer to the parenthesized expressions in the pattern, and are the equivalent of the \1, \2 that you might be familiar with if you've used grep. For example:
Here we've looked for five patterns, separated by any string starting with a comma and ending with the word curious. $0 stores the matched expression, $1 stores the first parenthesized sub-expression, and so on.
If you prefer to use quotation marks rather than the braces I used here, remember that certain characters have to be escaped with a backslash.

### The replacement expression

The previous example demonstrates that an important feature of replace is that the replacement doesn't have to be just a simple string or list, it can be any newLISP expression. Each time the pattern is found, the replacement expression is evaluated. You can use this to provide a replacement value that's calculated dynamically, or you could do anything else you wanted to with the found text. It's even possible to evaluate an expression that's got nothing to do with found text at all.
Here's another example: search for the letter t followed either by the letter h or by any vowel, and print out the combinations that replace found:
For every matching piece of text found, the third expression
was evaluated. This is a good way of seeing what the regular expression engine is up to while the function is running. In this example, the original string appears to be unchanged, but in fact it did change, because (println $0) did two things: it printed the string, and it returned the value to replace, thus replacing the found text with itself. Invisible mending! If the replacement expression doesn't return a string, no replacement occurs.
You could do other useful things too, such as build a list of matches for later processing, and you can use the newLISP system variables and any other function to use any of the text that was found.
In the next example, we look for the letters a, e, or c, and force each occurrence to upper-case:
As another example, here's a simple search and replace operation that keeps count of how many times the letter 'o' has been found in a string, and replaces each occurrence in the original string with the count so far. The replacement is a block of expressions grouped into a single begin expression. This block is evaluated every time a match is found:
The output from println doesn't appear in the string; the final value of the entire begin expression is a string version of the counter, so that gets inserted into the string.
Here's yet another example of replace in action. Suppose I have a text file, consisting of the following:
I want to write a newLISP script that re-numbers the lines in multiples of 10, starting at 10, and aligns the text so that the equals signs line up, like this:
(I don't know what language this is!)
The following script will do this:
I've used two replace operations inside the while loop, to keep things clearer. The first one sets a temporary variable to the result of a replace operation. The search string ({^(\d*)(\s*)(.*)}) is a regular expression that's looking for any number at the start of a line, followed by some space, followed by anything. The replacement string ((string (inc counter 10) " " $3) 0)) consists of a incremented counter value, followed by the third match (ie the anything I just looked for).
The result of the second replace operation is printed. I'm searching the temporary variable temp for more strings and spaces with an equals sign in the middle:
The replacement expression is built up from the important found elements ($1, $3, $5) but it also includes a quick calculation of the amount of space required to bring the equals sign across to character 20, which should be the difference between the first item's width and position 20 (which I've chosen arbitrarily as the location for the equals sign).
Regular expressions aren't very easy for the newcomer, but they're very powerful, particularly with newLISP's replace function, so they're worth learning.

## Testing and comparing strings

There are various tests that you can run on strings. newLISP's comparison operators work by finding and comparing the code numbers of the characters until a decision can be made:
and of course newLISP's flexible argument handling lets you test loads of strings at the same time:
These comparison functions also let you use them with a single argument. If you supply only one argument, newLISP helpfully assumes that you mean 0 or "", depending on the type of the first argument:
To check whether two strings share common features, you can either use starts-with and ends-with, or the more general pattern matching commands member, regex, find, and find-all. starts-with and ends-with are simple enough:
They can also accept regular expressions, using one of the regex options (0 being the most commonly used):
find, find-all, member, and regex look everywhere in a string. find returns the index of the matching substring:
member looks to see if one string is in another. It returns the rest of the string, including the search string, rather than the index of the first occurrence.
Both find and member let you use regular expressions:
find-all works like find, but returns a list of all matching strings, rather than the index of just the first match. It always takes regular expressions, so - for once - you don't have to put regex option numbers at the end.
Or you could use regex. This returns nil if the string doesn't contain the pattern, but, if it does contain the pattern, it returns a list with the matched strings and substrings, and the start and length of each string. The results can be quite complicated:
This results list can be interpreted as 'the first match was from character 0 continuing for 70 characters, the second from character 0 continuing for 15 characters, another from character 15 for 33 characters', and so on.
The matches are also stored in the system variables ($0, $1, ...) which you can inspect easily with a simple loop:

## Strings to lists

Two functions let you convert strings to lists, ready for manipulation with newLISP's extensive list-processing powers. The well-named explode function cracks open a string and returns a list of single characters:
The explosion is easily reversed with join. explode can also take an integer. This defines the size of the fragments. For example, to divide up a string into cryptographer-style 5 letter groups, remove the spaces and use explode like this:
You can do similar tricks with find-all. Watch the end, though:

## Parsing strings

parse is a powerful way of breaking strings up and returning the pieces. Used on its own, it breaks strings apart, usually at word boundaries, eats the boundaries, and returns a list of the remaining pieces:
Or you can supply a delimiting character, and parse breaks the string whenever it meets that character:
By the way, I could eliminate that first empty string from the list by filtering it out:
You can also specify a delimiter string rather than a delimiter character:
Best of all, though, you can specify a regular expression delimiter. Make sure you supply the options flag (0 or whatever), as with most of the regex functions in newLISP:
Here's that well-known quick and not very reliable HTML-tag stripper:
For parsing XML strings, newLISP provides the function xml-parse. See Working with XML.
Take care when using parse on text. Unless you specify exactly what you want, it thinks you're passing it newLISP source code. This can produce surprising results:
The semicolon is considered a comment character in newLISP, so parse has ignored it and everything that followed on that line. Tell it what you really want, using delimiters or regular expressions:
or
If you want to chop strings up in other ways, consider using find-all, which returns a list of strings that match a pattern. If you can specify the chopping operation as a regular expression, you're in luck. For example, if you want to split a number into groups of three digits, use this technique:
The pattern has to consider cases where there are 2 or 1 digits left over at the end.
parse eats the delimiters once they've done their work - find-all finds things and returns what it finds.

## Other string functions

There are other functions that work with strings. search looks for a string inside a file on disk:
This example looks in system.log for the string kernel. If it's found, newLISP rewinds the file pointer by 64 characters, then prints out three lines, showing the line in context.
There are also functions for working with base64-encoded files, and for encrypting strings.

## Formatting strings

It's worth mentioning the format function, which lets you insert the values of newLISP expressions into a pre-defined template string. Use %s to represent the location of a string expression inside the template, and other % codes to include numbers. For example, suppose you want to display a list of files like this:
A suitable template for folders (directories) looks like this:
Give the format function a template string, followed by the expression (f) that produces a file or folder name:
When this is evaluated, the contents of f is inserted into the string where the %s is. The code to generate a directory listing in this format, using the directory function, looks like this:
I'm using the directory? function to choose the right template string. A typical listing looks like this:
There are lots of formatting codes that you use to produce the output you want. You use numbers to control the alignment and precision of the strings and numbers. Just make sure that the % constructions in the format string match the expressions or symbols that appear after it, and that there are the same number of each.
Here's another example. We'll display the first 400 or so Unicode characters in decimal, hexadecimal, and binary. We'll use the bits function to generate a binary string. We feed a list of three values to format after the format string, which has three entries:

## Strings that make newLISP think

Lastly, I must mention eval and eval-string. Both of these let you give newLISP code to newLISP for evaluation. If it's valid newLISP, you'll see the result of the evaluation. eval wants an expression:
eval-string wants a string:
This means that you can build newLISP code, using any of the functions we've met, and then have it evaluated by newLISP. eval is particularly useful when you're defining macros - functions that delay evaluation until you choose to do it. See Macros.
You could use eval and eval-string to write programs that write programs.
The following curious piece of newLISP continually and mindlessly rearranges a few strings and tries to evaluate the result. Unsuccessful attempts are safely caught. When it finally becomes valid newLISP, it will be evaluated successfully and the result will satisfy the finishing condition and finish the loop.
I've used programs that were obviously written using this programming technique...

# Apply and map: applying functions to lists
## Making functions and data work together

Often, you'll find that you've got some data stored in a list and you want to apply a function to it. For example, suppose that you've acquired some temperature readings from a space probe, and they're stored in a list called data:
How are you going to add these numbers up, and then divide by the total, to find the average? Perhaps you think you could use add, which totals a list of floating-point numbers, but you're not working interactively, so you can't edit the code to read like this:
Since we're holding the data in a symbol called data, we might try this:
but no, this doesn't work, because add wants numbers to add, not a list. You could of course do it the hard way and write a loop that works through each item in the list and increases a running total each time:
This works fine. But newLISP has a much more powerful solution, for this and many other problems: you can treat functions as data and data as functions, so you can manipulate functions as easily as you can manipulate your data. You can just 'introduce' add and the data list to each other, and then stand back and let them get on with it.
There are two important functions for doing this: apply and map.

### apply

apply takes a function and a list, and makes them work together:
and this produces the required result. Here we've treated the add function like any other newLISP list, string, or number, using it as an argument to another function. You don't need to quote it (although you can), because apply is already expecting the name of a function.

### map

The other function that can make functions and lists work together is map, which applies a function to each item of a list, one by one. For example, if you wanted to apply the floor function to each element of the data list (to round them down to the nearest integer) you could combine map, floor, and the data as follows:
and the floor function is applied to each element of the data. The results are combined and returned in a new list.

### apply and map in more detail

Both apply and map let you treat functions as data. They have the same basic form:
where f is the name of a function and l is a list. The idea is that you tell newLISP to process a list using the function you specify.
The apply function uses all the elements in the list as arguments to the function, and evaluates the result.
Here, apply looks at the list, which in this case consists of a single string, and feeds the elements to the function as arguments. The string gets reversed. Notice that you don't have to quote the function but you do have to quote the list, because you don't want newLISP to evaluate it before the designated function gets a chance to consume it.
The map function, on the other hand, works through the list, element by element, like a sergeant major inspecting a row of soldiers, and applies the function to each element in turn, using the element as the argument. However, map remembers the result of each evaluation as it goes, collects them up, and returns them in a new list.
So map looks like a control-flow word, a bit like dolist, whereas apply is a way of controlling the newLISP list evaluation process from within your program, calling a function when and where you want it called, not just as part of the normal evaluation process.
If we adapt the previous example for map, it gives a similar result, although the result is a list, not a string:
Because we've used a list with only one element, the result is almost identical to the apply example, although notice that map returns a list whereas, in this example, apply returns a string:
The string has been extracted from the list, reversed, and then stored in another list created by map.
In the next example:
you can clearly see that map has applied reverse to each element of the list in turn, and returned a list of the resulting reversed strings.

## Write one in terms of the other?

To illustrate the relationship between these two functions, here is an attempt at defining map in terms of apply:
We're pushing the result of applying a function f to each list item to the end of a temporary list, and then relying on push returning the list at the end, just as map would do. This works, at least for simple expressions:
This example illustrates why map is so useful. It's an easy way to transform all the elements of a list without the hassle of working through them element by element using a dolist expression.

## More tricks

Both map and apply have more tricks up their sleeves. map can traverse more than one list at the same time. If you supply two or more lists, newLISP interleaves the elements of each list together, starting with the first elements of each list, and passes them as arguments to the function:
Here the first element of each list is passed as a pair to append, followed by the second element of each list, and so on.
This weaving together of strands is a bit like knitting with lists. Or like doing up a zip.
apply has a trick too. A third argument indicates how many of the preceding list's arguments the function should use. So if a function takes two arguments, and you supply three or more, apply comes back and makes another attempt, using the result of the first application and another argument. It continues eating its way through the list until all the arguments are used up.
To see this in action, let's first define a function that takes two arguments and compares their lengths:
Now you can apply this function to a list of strings, using the third argument to tell apply to use up the arguments two strings at a time:
It's like walking along the beach and finding a pebble, and holding on to it until an even better one turns up.
apply also gives you a way of working through a list and applying a function to each pair of items:
What's happening here is that the value returned by the println function is the second member of the pair, and this becomes the value of the first element of the next pair.

## Lispiness

This thing about passing around the names of functions as if they were bits of data is very characteristic of newLISP, and it's very useful. You will find many uses for it, sometimes using functions that you don't think will be useful with map. Here, for example, is set working hard under the control of map:
This construction gives you another way to assign values to symbols in parallel, rather than sequentially. (You can use swap as well.)
Some uses of map are simple:
Others can become quite complex. For example, given a string of data in this form, stored in a symbol image-data:
the two numbers can be extracted with:

## currying

Some of the built-in newLISP functions do things with other functions. An example is curry, which creates a copy of a two-argument function and creates a single-argument version with a pre-determined first argument. So if a function f1 was often called like this:
you can use curry to make a new function f2 that has a ready-to-use built-in arg1:
now you can forget about that first argument, and just supply the second one to f2:
Why is this useful? Consider the dup function, which often gets used to insert multiple blank spaces:
Using curry, you can create a new function called, say, blank, that's a special version of dup that always gets called with a blank space as the string:
Now you can use (blank n):
curry can be useful for creating temporary or anonymous functions with map:
But avoid using curry on destructive functions like inc, for example:

# Introducing contexts

We all like to organize our stuff into separate areas or compartments. Chefs keep their fish, meat, and dessert areas separate, electronics engineers keep their power supplies away from their radio frequency and audio stages, and newLISP programmers use contexts to organize their code.

## What is a context?

A newLISP context provides a named container for symbols. Symbols in different contexts can have the same name without clashing. So, for example, in one context I can define the symbol called meaning-of-life to have the value 42, but, in another context, the identically-named symbol could have the value dna-propagation, and, in yet another, worship-of-deity.
Unless you specifically choose to create and/or switch contexts, all your newLISP work is carried out in the default context, called MAIN. So far in this document, when new symbols have been created, they've been added to the MAIN context.
Contexts are very versatile - you can use them for dictionaries, or software objects, or super-functions, depending on the task in hand.

## Contexts: the basics

The context function can be used for a number of different tasks:
* to create a new context
* to switch from one context to another
* to retrieve the value of an existing symbol in a context
* to see what context you're in
* to create a new symbol in a context and assign a value to it
newLISP can usually read your mind, and knows what you want to do, depending on how you use the context function. For example:
creates a new context called Test, as you might expect. If you type this in interactively, you'll see that newLISP changes the prompt to tell you that you're now working in another context:
And you can switch between contexts freely:
Used on its own, it just tells you where you are:
Once a context exists, you don't have to quote the name (but you can if you like). Notice that I've used an upper-case letter for my context name. This is not compulsory, just a convention.
A context contains symbols and their values. There are various ways to create a symbol and give it a value.
This creates a new context - notice the quote, because newLISP hasn't seen this before - and a new symbol called "villain", with a value of "Moriarty", but stays in the MAIN context. If the context already exists, you can omit the quote:
To obtain the value of a symbol, you can do this:
or, if you're using the console, this step by step approach:
or, from the MAIN context:
The full address of a symbol is the context name, followed by a colon (:), followed by the symbol name. Always use the full address if you're in another context.
To see all the symbols inside a context, use symbols to produce a list:
or, if you're already inside the Doyle context:
You can use this list of symbols in the usual way, such as stepping through it with dolist.
To see the values of each symbol, use eval to find its value, and term to return just the symbol's name.
There's a more efficient (slightly faster) technique for looping through symbols in a context. Use the dotree function:

### Creating contexts implicitly

As well as explicitly creating contexts with context, you can have newLISP create contexts for you automatically. For example:
Here, newLISP has created a new context C and a function called greeting in that context. You can create symbols this way too:
In both these examples, notice that you stayed in the MAIN context.
The following code creates a new context L containing a new list called ls that contains strings:

#### Functions in context

Contexts can contain functions and symbols. To create a function in a context other than MAIN, either do this:
or do this
This second syntax lets you create both the context and the function inside the context, while remaining safely in the MAIN context all the time.
You don't have to quote the new context name here because we're using define, and define (by definition) isn't expecting the name of an existing symbol.
To use functions while you're in another context, remember to call them using this context:function syntax.

## The default function

If a symbol in a context has the same name as the context, it's known as the default function (although in fact it can be either a function, or a symbol containing a list or a string). For example, here is a context called Evens, and it contains a symbol called Evens:
And here is a function called Double in a context called Double:
So Evens and Double are the default functions for their contexts.
There are lots of good things about default functions. If the default function has the same name as the context, it is evaluated whenever you use the name of the context in expressions, unless newLISP is expecting the name of a context. For example, although you can always switch to the Evens context by using the context function in the usual way:
you can use Evens as a list (because Evens:Evens is a list):
You can use the default function without supplying its full address. Similarly, you can use the Double function as an ordinary function without supplying the full colon-separated address:
You can still switch to the Double context in the usual way:
newLISP is smart enough to be able to work out from your code whether to use the default function of a context or the context itself.

### Passing parameters by reference

There are important differences between default functions when used as symbols and their more ordinary siblings. When you use a default function to pass data to a function, newLISP uses a reference to the data rather than a copy. For larger lists and strings, references are much quicker for newLISP to pass around between functions, so your code will be faster if you can use store data as a default function and use the context name as a parameter.
Also, and as a consequence, functions change the contents of any default functions passed as reference parameters. Ordinary symbols are copied when passed as parameters. Observe the following code. I'll create two symbols, one of which is a 'default function', the other is a plain symbol:
So far, it looks as if they're behaving identically. But now inspect the original symbols:
The list that was passed as a default function - as a reference - was modified, whereas the ordinary list parameter was copied, as usual, and wasn't modified.

### Functions with a memory

In the following example, we create a context called Output, and a default function inside it, also called Output. This function prints its arguments, and increments a counter by the number of characters output. Because the default function has the same name as the context, it is executed whenever we use the name of the context in other expressions.
Inside this function, the value of a variable counter (inside the context Output) is incremented if it exists, or created and initialized if it doesn't. Then the function's main task - the printing of the arguments - is done. The counter symbol keeps count of how many characters were output.
The Output function effectively remembers how much work it's done since it was first created. It could even append that information to a log file.
Think of the possibilities. You could log the usage of all your functions, and bill users according to how often they use them.
You can override the built-in println function so that it uses this code instead when it's called. See On your own terms.

## Dictionaries and tables

A common use for a context is a dictionary: an ordered set of unique key/value pairs, arranged so that you can obtain the current value of a key, or add a new key/value pair. newLISP makes creating dictionaries easy. To illustrate, I'll enlist the help of the great detective, Sherlock Holmes. First, I downloaded Sir Arthur Conan Doyle's The Sign of Four from Project Gutenberg, then I loaded the file as a list of words.
Next, I define an empty dictionary:
This defines the Doyle context and the default function, but leaves that default function uninitialized. If the default function is left empty, you can use the following expressions to build and examine a dictionary:
* (Doyle key value) - set key to value
* (Doyle key) - get value of key
* (Doyle key nil) - delete key
To build a dictionary from the word list, you scan through the words, and, if the word is not in the dictionary, add it as the key, and set the value to 1. But if the word is already in the dictionary, get the value, add 1 to it, and save the new value:
This shorter alternative eliminates the conditional:
or, even shorter still:
Each word is added to the dictionary, and the value (the number of occurrences) increased by 1. Inside the context, the names of the keys have been prefixed with an underscore ("_"). This is so that nobody gets confused between the names of keys and newLISP reserved words, many of which occur in Conan-Doyle's text.
There are various ways you can browse the dictionary. To look at individual symbols:
To look at the symbols as they are stored in a context, work through the context evaluating each symbol, using dotree:
To see the dictionary as an association list, use the dictionary name on its own. This creates a new association list::
This is a standard association list, which you can access using the functions described in the Lists chapter (see Association lists). For example, to find all words that occur 20 times, use find-all:
The association list returned by (Doyle) is a temporary copy of the data in the dictionary, not the original dictionary context. To change the data, don't operate on this temporary list but on the context's data, using the key/value access techniques.
You can also add new entries to dictionaries, or modify existing entries, using data in the form of an association list:

## Saving and loading contexts

If you want to use the dictionary again, you can save the context in a file:
This collection of data, wrapped up in a context called Doyle, can be quickly loaded by another script or newLISP session using:
and newLISP will automatically recreate all the symbols in the Doyle context, switching back to the MAIN (default) context when done.

## Using newLISP modules

Contexts are used as containers for software modules because they provide lexically-separated namespaces. The modules supplied with the newLISP installation usually define a context that contains a set of functions handling tasks in a specific area.
Here's an example. The POP3 module lets you check POP3 email accounts. You first load the module:
The module has now been added to the newLISP system. You can switch to the context:
and call the functions in the context. For example, to check your email, use the get-mail-status function, supplying user name, password, and POP3 server name:
If you don't switch to the context, you can still call the same function by supplying the full address:

## Scoping

You've already seen the way newLISP dynamically finds the current version of a symbol (see Scope). However, when you use contexts, you can employ a different approach, which programmers call lexical scoping. With lexical scoping, you can explicitly control which symbol is used, rather than rely on newLISP to keep track of similarly-named symbols for you automatically.
In the following code, the width symbol is defined inside the Right-just context.
The second (set 'width ...) line is a red herring: changing this here makes no difference at all, because the symbol which is actually used by the right-justification function is inside a different context.
You can still reach inside the Right-just context to set the width:
There's been much discussion about the benefits and disadvantages of the two approaches. Whatever you choose, make sure you know where the symbols are going to get their values from when the code runs. For example:
Here, y is the first argument to the function, and is independent of any other y. But what about x? Is it a global symbol, or has the value been defined in some other function that has just called f? Or perhaps it has no value at all!
It's best to avoid using these free symbols, and to use local variables (defined with let or local) wherever possible. Perhaps you can adopt a convention such as putting asterisks around a global symbol.

## Objects

More has been written about object-oriented programming (OOP) than you could possibly read in one lifetime, so this section is just a quick glance at the subject. newLISP is agile enough to enable more than one style of OOP, and you can easily find references to these on the web, together with discussions about the merits of each.
For this introduction, I'll briefly outline just one of these styles: FOOP, or Functional Object-Oriented Programming.

### FOOP in a nutshell

FOOP has changed with newLISP version 10.2 (beginning of 2010), so if you're using and old version of newLISP, please update it.
In FOOP, each object is stored as a list. Class methods and class properties (ie functions and symbols that apply to every object of that class) are stored in a context.
Objects are stored in lists because lists are fundamental to newLISP. The first item in an object list is a symbol identifying the class of the object; the remaining items are the values that describe the properties of an object.
All the objects in a class share the same properties but those properties can have different values. The class can also have properties that are shared between all objects in the class; these are the class properties. Functions stored in the class context provide the various methods for managing the objects and processing the data they hold.
To illustrate these ideas, consider the following code that works with times and dates. It builds on top of the basic date and time functions provided by newLISP (see Working with dates and times). A moment in time is represented as a time object. An object holds two values: the number of seconds that have elapsed since the beginning of 1970, and the time zone offset, in minutes west of Greenwich. So the list to represent a typical time object looks like this:
where Time is a symbol representing the class name, and the two numbers are the values of this particular time (these numbers represent a time around 10 in the morning on Sunday August 24 2008, somewhere in England).
The code required to build this object is straightforward using newLISP generic FOOP constructor:
However, you might want to define a different constructor, for example, you might want to give this object some default values. To do that you have to redefine the default function which is acting as the constructor:
It's a default function for the Time context that builds a list with the class name in the first position, and two more integers to represent the time. When values are not supplied, they default to the current time and zero offset. You can now use the constructor without supplying some or any parameters:
Next, you can define other functions to inspect and manage time objects. All these functions live together in the context. They can extract the seconds and zone information by picking them from the object by using the self function. So (self 1) gets the seconds and (self 2) gets the time zone offset from the object passed as parameter. Notice the definitions do not require you to state the object parameter. Here are a few obvious class functions:
These functions are called by using the colon operator and providing the object which we want the function to act upon:
Notice how we used the colon operator as a prefix to the function, without separating it with a space, this is a matter of style, you can use it with ot without spaces:
This technique allows newLISP to provide a feature that object-oriented programmers love: polymorphism.

### Polymorphism

Let's add another class which works on durations - the gap between two time objects - measured in days.
There's a new class constructor Duration:Duration for making new duration objects, and a simple show function. They can be used with the time objects like this:
Compare that :show function call with the :show in the previous section:
You can see that newLISP is choosing which version of the show function to evaluate, according to the class of :show's parameter. Because christmas-day is a Time object, newLISP evaluates Time:show. But when the argument is a Duration object, it evaluates Duration:show. The idea is that you can use a function on various types of object: you perhaps don't need to know what class of object you're dealing with. With this polymorphism, you can apply the show function to a list of objects of differing types, and newLISP selects the appropriate one each time:
Note: We have to use curry here because map needs to use both the colon operator as well as the show function.

### Modifying objects

We're calling this particular style of OOP FOOP because it's considered to be functional. Here, the term 'functional' refers to the style of programming encouraged by newLISP which emphasizes the evaluation of functions, avoiding state and mutable data. As you've seen, many newLISP functions return copies of lists rather than modify the originals. There are, though, a small number of functions that are called destructive, and these are considered to be less purely functional. But FOOP doesn't provide for destructive object methods, so can be considered more functional.
A key point to notice about FOOP is that objects are immutable; they can't be modified by class functions. For example, here's a function for the Time class that adds a given number of days to a time object:
When this is called, it returns a modified copy of the object; the original is unchanged:
The original date of the christmas-day object didn't change, although the :adjust-days function returned a modified copy adjusted by 3 days.
In other words, to make changes to objects, use the familiar newLISP approach of using the value returned by a function:
christmas-day now contains the modified date.
You can find a more complete working out of this idea by searching the newLISP forum for timeutilities. Also make sure to read the section on FOOP in the reference manual, it has a nice example on nested objects, that is, objects containing other objects.

# Macros
## Introducing macros

We've covered the basics of newLISP, but there are plenty of powerful features left to discover. Once you've grasped the main rules of the language, you can decide which of the more advanced tools you want to add. One feature that you might want to explore is newLISP's provision of macros.
A macro is a special type of function that you can use to change the way your code is evaluated by newLISP. For example, you can create new types of control functions, such as your own version of if or case.
With macros, you can start to make newLISP work exactly the way you want it to.
Strictly speaking, newLISP's macros are fexprs, not macros. In newLISP, fexprs are called macros partly because it's much easier to say 'macros' than 'fexprs' but mainly because they serve a similar purpose to macros in other LISP dialects: they allow you to define special forms such as your own control functions.

## When do things get evaluated

To understand macros, let's jump back to one of the very first examples in this introduction. Consider the way this expression is evaluated:
The * function doesn't see the + expressions at all, only their results. newLISP has enthusiastically evaluated the addition expressions and then handed just the results to the multiplication function. This is usually what you want, but there are times when you don't want every expression evaluated immediately.
Consider the operation of the built-in function if:
If x is greater than 0, the test returns nil so the (exit) function isn't evaluated. Now suppose that you want to define your own version of the if function. It ought to be easy:
But this doesn't work. If the comparison returns true, newLISP prints a message and then exits. Even if the comparison returned false, newLISP would still exit, without printing a message. The problem is that (exit) is evaluated before the my-if function is called, even when you don't want it to be. For ordinary functions, expressions in arguments are evaluated first.
Macros are similar to functions, but they let you control when - and if - arguments are evaluated. You use the define-macro function to define macros, in the same way that you define functions using define. Both these defining functions let you make your own functions that accept arguments. The important difference is that, with the ordinary define, arguments are evaluated before the function runs. But when you call a macro function defined with define-macro, the arguments are passed to the definition in their raw and unevaluated form. You decide when the arguments are evaluated.
A macro version of the my-if function looks like this:
The test and action arguments aren't evaluated immediately, only when you want them to be, using eval. And this means that the (exit) isn't evaluated before the test has been made.
This ability to postpone evaluation gives you the ability to write your own control structures and add powerful new forms to the language.

## Tools for building macros

newLISP provides a number of useful tools for building macros. As well as define-macro and eval, there's letex, which gives you a way of expanding local symbols into an expression before evaluating it, and args, which returns all the arguments that are passed to your macro.

## Symbol confusion

One problem to be aware of when you're writing macros is the way that symbol names in macros can be confused with symbol names in the code that calls the macro. Here's a simple macro which adds a new looping construct to the language that combines dolist and do-while. A loop variable steps through a list while a condition is true:
It's called like this:
And it appears to work well. But there's a subtle problem: you can't use a symbol called y as the loop variable, even though you can use x or anything else. Put a (println y) statement in the loop to see why:
If you try to use y, it won't work:
The problem is that y is used by the macro to hold the condition value, even though it's in its own let expression. It appears as a true/nil value, so it can't be decremented. To fix this problem, enclose the macro inside a context, and make the macro the default function in that context:
This can be used in the same way, but without any problems:

#### Other ideas for macros

newLISP users find many different reasons to use macros. Here are a couple of macro definitions I've found on the newLISP user forums.
Here's a version of case, called ecase (evaluated-case) that really does evaluate the tests:
You can see that the divisions (/ 4 4) and (- 12 10) were both evaluated. They wouldn't have been with the standard version of case.
Here's a macro that creates functions:

### A tracer macro

The following code changes the operation of newLISP so that every function defined using define will, when evaluated, add its name and details of its arguments to a log file. When you run a script, the log file will contain a record of the functions and arguments that were evaluated.
To run a script with this simple tracer, load the context before you run:
The log file generated contains a list of every function that was called, and the arguments it received:
It will slow execution quite a lot.

# Working with numbers

If you work with numbers, you'll be pleased to know that newLISP includes most of the basic functions that you'd expect to find, plus many more. This section is designed to help you make the best use of them and to avoid some of the minor pitfalls that you might encounter. As always, see the official documentation for full details.

## Integers and floating-point numbers

newLISP handles two different types of number: the integer and the floating-point number. Integers are precise, whereas floating-point numbers (floats) are less precise. There are advantages and disadvantages to each. If you need to use very large integers, larger than 9 223 372 036 854 775 807, see the section covering the differences between large (64-bit) integers and big integers (of unlimited size) - Bigger numbers.
The arithmetic operators +, -, *, /, and % always return integer values. A common mistake is to forget this and use / and * without realising that they're carrying out integer arithmetic:
This might not be what you were expecting!
Floating-point numbers keep only the 15 or 16 most important digits (ie the digits at the left of the number, with the highest place values).
The philosophy of a floating-point number is that's close enough, rather than that's the exact value.
Suppose you try to define a symbol PI to store the value of pi to 50 decimal places:
It looks like newLISP has cut about 40 digits off the right hand side! In fact about 15 or 16 digits have been stored, and 35 of the less important digits have been discarded.
How does newLISP store this number? Let's look using the format function:
Now let's make a little script to compare both numbers as strings, so we don't have to grep visually the differences:
Notice how the value is accurate up to 9793, but then drifts away from the more precise string you originally supplied. The numbers after 9793 are typical of the way all computers store floating-point values - it isn't newLISP being creative with your data!
The largest float you can use seems to be - on my machine, at least - about 10$^{308}$. Only the first 15 or so digits are stored, though, so that's mostly zeroes, and you can't really add 1 to it.
Another example of the motto of a floating-point number: that's close enough!
The above comments are true for most computer languages, by the way, not just newLISP. Floating-point numbers are a compromise between convenience, speed, and accuracy.

## Integer and floating-point maths

When you're working with floating-point numbers, use the floating-point arithmetic operators add, sub, mul, div, and mod, rather than +, -, *, /, and %, their integer-only equivalents:
and, to see the value that newLISP is storing (because the interpreter's default output resolution is 9 or 10 digits):
If you forget to use mul here, and use * instead, the numbers after the decimal point are thrown away:
Here, pi was converted to 3 and then multiplied by 2.
You can re-define the familiar arithmetic operators so that they default to using floating-point routines rather than integer-only arithmetic:
You could put these definitions in your init.lsp file to have them available for all newLISP work on your machine. The main problem you'll find is when sharing code with others, or using imported libraries. Their code might produce surprising results, or yours might!

## Conversions: explicit and implicit

To convert strings into numbers, or numbers of one type into another, use the int and float functions.
The main use for these is to convert a string into a number - either an integer or a float. For example, you might be using a regular expression to extract a string of digits from a longer string:
A second argument passed to int specifies a default value which should be used if the conversion fails:
int is a clever function that can also convert strings representing numbers in number bases other than 10 into numbers. For example, to convert a hexadecimal number in string form to a decimal number, make sure it is prefixed with 0x, and don't use letters beyond f:
And you can convert strings containing octal numbers by prefixing them with just a 0:
Binary numbers can be converted by prefixing them with 0b:
Even if you never use octal or hexadecimal, it's worth knowing about these conversions, because one day you might, either deliberately or accidentally, write this:
which evaluates to 0 rather than 8 - a failed octal-decimal conversion rather than the decimal 8 that you might have expected! For this reason, it's always a good idea to specify not only a default value but also a number base whenever you use int on string input:
If you're working with big integers (integers larger than 64-bit integers), use bigint rather than int. See Bigger numbers.

## Invisible conversion and rounding

Some functions convert floating-point numbers to integers automatically. Since newLISP version 10.2.0 all operators made of letters of the alphabet produce floats and operators written with special characters produce integers.
So using ++ will convert and round your numbers to integers, and using inc will convert your numbers to floats:
To make inc and dec work on lists you need to access specific elements or use map to process all:
Many newLISP functions automatically convert integer arguments into floating-point values. This usually isn't a problem. But it's possible to lose some precision if you pass extremely large integers to functions that convert to floating-point:
Because the add function converted the very large integer to a float, a small amount of precision was lost (amounting to about 52, in this case). Close enough? If not, think carefully about how you store and manipulate numbers.

## Number testing

Sometimes you will want to test whether a number is an integer or a float:
With integer? and float?, you're testing whether the number is stored as an integer or float, not whether the number is mathematically an integer or a floating-point value. For example, this test returns nil, which might surprise you:
It's not that the answer isn't 10 (it is), but rather that the answer is a floating-point 10, not an integer 10, because the div function always returns a floating-point value.

## Absolute signs, from floor to ceiling

It's worth knowing that the floor and ceil functions return floating-point numbers that contain integer values. For example, if you use floor to round pi down to the nearest integer, the result is 3, but it's stored as a float not as an integer:
The abs and sgn functions can also be used when testing and converting numbers. abs always returns a positive version of its argument, and sgn returns 1, 0, or -1, depending on whether the argument is positive, zero, or negative.
The round function rounds numbers to the nearest whole number, with floats remaining floats. You can also supply an optional additional value to round the number to a specific number of digits. Negative numbers round after the decimal point, positive numbers round before the decimal point.
sgn has an alternative syntax that lets you evaluate up to three different expressions depending on whether the first argument is negative, zero, or positive.

## Number formatting

To convert numbers into strings, use the string and format functions:
Both string and println use only the first 10 or so digits, even though more (up to 15 or 16) are stored internally.
Use format to output numbers with more control:
The format specification string uses the widely-adopted printf-style formatting. Remember too that you can use the results of the format function:
The format function lets you output numbers as hexadecimal strings as well:

## Number utilities
### Creating numbers

There are some useful functions that make creating numbers easy.

## Sequences and series

sequence produces a list of numbers in an arithmetical sequence. Supply start and finish numbers (inclusive), and a step value:
If you specify a step value, all the numbers are stored as floats, even if the results are integers, otherwise they're integers:
series multiplies its first argument by its second argument a number of times. The number of repeats is specified by the third argument. This produces geometric sequences:
Every number is stored as a float.
The second argument of series can also be a function. The function is applied to the first number, then to the result, then to that result, and so on.
The normal function returns a list of floating-point numbers with a specified mean and a standard deviation. For example, a list of 6 numbers with a mean of 10 and a standard deviation of 5 can be produced as follows:

## Random numbers

rand creates a list of randomly chosen integers less than a number you supply:
Obviously (rand 1) generates a list of zeroes and isn't useful. (rand 0) doesn't do anything useful either, but it's been assigned the job of initializing the random number generator.
If you leave out the second number, it just generates a single random number in the range.
random generates a list of floating-point numbers multiplied by a scale factor, starting at the first argument:

## Randomness

Use seed to control the randomness of rand (integers), random (floats), randomize (shuffled lists), and amb (list elements chosen at random).
If you don't use seed, the same set of random numbers appears each time. This provides you with a predictable randomness - useful for debugging. When you want to simulate the randomness of the real world, seed the random number generator with a different value each time you run the script:
Without seed:
With seed:

### General number tools

min and max work as you would expect though they always return floats. Like many of the arithmetic operators, you can supply more than one value:
The comparison functions allow you to supply just a single argument. If you use them with numbers, newLISP helpfully assumes that you're comparing with 0. Remember that you're using postfix notation:
The factor function finds the factors for an integer and returns them in a list. It's a useful way of testing a number to see if it's prime:
Or you could use it to test if a number is even:
gcd finds the largest integer that exactly divides two or more numbers:

### Floating-point utilities

If omitted, the second argument to the pow function defaults to 2.
You can also use sqrt to find square roots. To find cube and other roots, use pow:
The exp function calculates e$^x$, where e is the mathematical constant 2.718281828, and x is the argument:
The log function has two forms. If you omit the base, natural logarithms are used:
Or you can specify another base, such as 2 or 10:
Other mathematical functions available by default in newLISP are fft (fast Fourier transform), and ifft (inverse fast Fourier transform).

## Trigonometry

All newLISP's trigonometry functions, sin, cos, tan, asin, acos, atan, atan2, and the hyperbolic functions sinh, cosh, and tanh, work in radians. If you prefer to work in degrees, you can define alternative versions as functions:
and so on.
When writing equations, one approach is to build them up from the end first. For example, to convert an equation like this:
$\alpha =\arctan \left\{{\frac {\sin \lambda \cos \epsilon -\tan \beta \sin \epsilon }{\cos \lambda }}\right\}$
build it up in stages, like this:
and so on...
It's often useful to line up the various expressions in your text editor:
If you have to convert a lot of mathematical expressions from infix to postfix notation, you might want to investigate the infix.lsp module (available from the newLISP website):

## Arrays

newLISP provides multidimensional arrays. Arrays are very similar to lists, and you can use most of the functions that operate on lists on arrays too.
A large array can be faster than a list of similar size. The following code uses the time function to compare how fast arrays and lists work.
The exact times will vary from machine to machine, but typically, with 200 elements, arrays and lists are comparable in speed. As the sizes of the list and array increase, the execution time of the nth accessor function increases. By the time the list and array contain 1000 elements each, the array is 2 to 3 times faster to access than the list.
To create an array, use the array function. You can make a new empty array, make a new one and fill it with default values, or make a new array that's an exact copy of an existing list.
To make a new list that's a copy of an existing array, use the array-list function:
To tell the difference between lists and arrays, you can use the list? and array? tests:

### Functions available for arrays

The following general-purpose functions work equally well on arrays and lists: first, last, rest, mat, nth, setf, sort, append, and slice.
There are also some special functions for arrays and lists that provide matrix operations: invert, det, multiply, transpose. See Matrices.
Arrays can be multi-dimensional. For example, to create a 2 by 2 table, filled with 0s, use this:
The third argument to array supplies some initial values that newLISP will use to fill the array. newLISP uses the value as effectively as it can. So, for example, you can supply a more than sufficient initializing expression:
or just provide a hint or two:
This array initialization facility is cool, so I sometimes use it even when I'm creating lists:

## Getting and setting values

To get values from an array, use the nth function, which expects a list of indices for the dimensions of the array, followed by the name of the array:
(nth also works with lists and strings.)
As with lists, you can use implicit addressing to get values:
To set values, use setf. The following code replaces every number that isn't prime with 0.
Instead of the implicit addressing (table row column), I could have written (setf (nth (list row column) table) 0). Implicit addressing is slightly faster, but using nth can make code easier to read sometimes.

## Matrices

There are functions that treat an array or a list (with the correct structure) as a matrix.
* invert returns the inversion of a matrix
* det calculates the determinant
* multiply multiplies two matrices
* mat applies a function to two matrices or to a matrix and a number
* transpose returns the transposition of a matrix
transpose is also useful when used on nested lists (see Association lists).

## Statistics, financial, and modelling functions

newLISP has an extensive set of functions for financial and statistical analysis, and for simulation modelling.
Given a list of numbers, the stats function returns the number of values, the mean, average deviation from mean value, standard deviation (population estimate), variance (population estimate), skew of distribution, and kurtosis of distribution:
Here's a list of other functions built in:
* beta calculate the beta function
* betai calculate the incomplete beta function
* binomial calculate the binomial function
* corr calculate the Pearson product-moment correlation coefficient
* crit-chi2 calculate the Chi square for a given probability
* crit-f calculate the critical minimum F for a given confidence probability
* crit-t calculate the critical minimum Student's t for a given confidence probability
* crit-z calculate the critical normal distributed Z value of a given cumulated probability
* erf calculate the error function of a number
* gammai calculate the incomplete gamma function
* gammaln calculate the log gamma function
* kmeans-query calculate the Euclidian distances from the data vector to centroids
* kmeans-train perform Kmeans cluster analysis on matrix-data
* normal produce a list of normal distributed floating point numbers
* prob-chi2 calculate the cumulated probability of a Chi square
* prob-f find the probability of an observed statistic
* prob-t find the probability of normal distributed value
* prob-z calculate the cumulated probability of a Z value
* stats find statistical values of central tendency and distribution moments of values
* t-test use student's t-test to compare the mean value

## Bayesian analysis

Statistical methods developed initially by Reverend Thomas Bayes in the 18th century have proved versatile and popular enough to enter the programming languages of today. In newLISP, two functions, bayes-train and bayes-query, work together to provide an easy way to calculate Bayesian probabilities for datasets.
Here's how to use the two functions to predict the likelihood that a short piece of text is written by one of two authors.
First, choose texts from the two authors, and generate datasets for each. I've chosen Oscar Wilde and Conan Doyle.
The bayes-train function can now scan these two data sets and store the word frequencies in a new context, which I'm calling Lexicon:
This context now contains a list of words that occur in the lists, and the frequencies of each. For example:
ie the word always appeared 21 times in Conan Doyle's text, and 110 times in Wilde's. Next, the Lexicon context can be saved in a file:
and reloaded whenever necessary with:
With training completed, you can use the bayes-query function to look up a list of words in a context, and return two numbers, the probabilities of the words belonging to the first or second set of words. Here are three queries. Remember that the first set was Doyle, the second was Wilde:
These numbers suggest that quote1 is probably (97% certain) from Conan Doyle, that quote2 is neither Doylean nor Wildean, and that quote3 is likely to be from Oscar Wilde.
Perhaps that was lucky, but it's a good result. The first quote is from Doyle's A Study in Scarlet, and the third is from Wilde's Lord Arthur Savile's Crime, both texts that were not included in the training process but - apparently - typical of the author's vocabulary. The second quote is from Jane Austen, and the methods developed by the Reverend are unable to assign it to either of the authors.

## Financial functions

newLISP offers the following financial functions:
* fv returns the future value of an investment
* irr returns the internal rate of return
* nper returns the number of periods for an investment
* npv returns the net present value of an investment
* pmt returns the payment for a loan
* pv returns the present value of an investment

## Logic programming

The programming language Prolog made popular a type of logic programming called unification. newLISP provides a unify function that can carry out unification, by matching expressions.
When using unify, unbound variables start with an uppercase character to distinguish them from symbols.

## Bit operators

The bit operators treat numbers as if they consist of 1's and 0's. We'll use a utility function that prints out numbers in binary format using the bits function:
This function prints out both the original number and a binary representation of it:
The shift functions (<< and >>) move the bits to the right or left:
The following operators compare the bits of two or more numbers. Using 4 and 5 as examples:
The negate or not function (~) reverses all the bits in a number, exchanging 1's and 0's:
The binary function that prints out these strings uses the & function to test the last bit of the number to see if it's a 1, and the >> function to shift the number 1 bit to the right, ready for the next iteration.
One use for the OR operator (|) is when you want to combine regular expression options with the regex function.
crc32 calculates a 32 bit CRC (Cyclic Redundancy Check) for a string.

## Bigger numbers

For most applications, integer calculations in newLISP involve whole numbers up to 9223372036854775807 or down to -9223372036854775808. These are the largest integers you can store using 64 bits. If you add 1 to the largest 64-bit integer, you'll 'roll over' (or wrap round) to the negative end of the range:
But newLISP can handle much bigger integers than this, the so-called 'bignums' or 'big integers'.
Notice that newLISP indicates a big integer using a trailing "L". Usually, you can do calculations with big integers without any thought:
Here both operands are big integers, so the answer is automatically big as well.
However, you need to take more care when your calculations combine big integers with other types of number. The rule is that the first argument of a calculation determines whether to use big integers. Compare this loop:
with this:
In the first example, the first argument of the function was a large (64-bit integer). So adding 1 to the largest possible 64 bit integer caused a roll-over - the calculation stayed in the large integer realm.
In the second example, the L appended to the first argument of the addition forced newLISP to switch to big integer operations even though both the operands were 64 bit integers. The size of the first argument determines the size of the result.
If you supply a literal big integer, you don't have to append the "L", since it's obvious that the number is a big integer:
There are other ways you can control the way newLISP converts between large and big integers. For example, you can convert something to a big integer using the bigint function:

# Working with dates and times
## Date and time functions

To work with dates and times, use the following functions:
* date convert a seconds count to a date/time, or return date/time for now
* date-value return the time in seconds since 1970-1-1 for a date and time, or for now
* now return the current date/time information in a list
* time-of-day return milliseconds since the start of today till now
date-value and now work in UT, not your local time. date can take account of the time difference between your local time and UT.

## The current time and date

All four functions can be used to return information about the current time. date-value returns the number of seconds between 1970 and the current time (in UT):
and now returns a list of integers containing information about the current date and time (in UT):
This provides the following information:
* year, month, day (2006, 3, 19)
* hour, minute, second, microsecond (20, 5, 2, 125475)
* day of current year (78)
* day of current week (1)
* local time zone offset (in minutes west of GMT) (0)
* daylight savings time flag (0)
To extract the information you want, use a slice or pick out the elements:
date used on its own will give you the current date and time for the local time zone (now and date-value return values in UCT/UTC, not relative to your local time zone):
It can also tell you the date of an integer number of seconds since 1970 (the start of the Unix epoch), adjusted for your local time zone:
date-value can calculate the number of seconds for a specific date or date/time (in UT):
Because date-value can accept the year, month, day, hours, minutes, and seconds as input, it can be applied to the output of now:
By converting different times to these date values, you can do calculations. For example, to subtract November 13 2003 from January 3 2005:
You can find out the date that is 12 days after Christmas Day 2005 by adding 12 days-worth of seconds to the date:
This seconds value can be converted to a human-friendly date by date in its longer form, when it takes a seconds-since-1970 value and converts it to a local time zone representation of this UT-based value:
Of course (date (date-value)) is the same as (date), but you'll have to use the longer form if you want to change the date format. date accepts an additional formatting string (preceded by a time-zone offset in minutes). If you're familiar with C-style strftime formatting, you'll know what to do:

### Reading dates and times: parse-date

The parse-date function (which is, unfortunately, not available on Windows) can convert date and time strings to seconds-since-1970 values. You supply a date-time format string after the string:

## Timing and timers

For timing purposes, you can use these functions:
* time return the time taken to evaluate an expression, in milliseconds
* timer set a timer to wait for a certain number of seconds and then evaluate expression
* sleep stop working for a certain number of milliseconds
time is useful for finding out how much time expressions take to evaluate:
You can supply a repetition count as well, which probably gives a more accurate picture:
If you can't or don't want to enclose expressions, more simple timing can be done using time-of-day:
timer is basically an alarm clock. Set it and then forget about it until the time comes. You supply a symbol specifying the alarm action, followed by the number of seconds to wait:
and three minutes later you'll see this:
Without any arguments, this function returns the name of the current symbol that's been assigned as the alarm action:
If you're waiting for the alarm to go off, and you're impatient to see how much time has elapsed so far, use the function with the name of the assigned symbol but without a seconds value:
For another example of the use of these functions, see Simple countdown timer.

# Working with files

Functions for working with files can be grouped into two main categories: interacting with the operating system, and reading and writing data to and from files.

## Interacting with the file system

The concept of a current working directory is maintained in newLISP. When you start newLISP by typing newLISP in a terminal, your current working directory becomes newLISP's current directory.
You can also check your current working directory using the real-path function without arguments:
But when you run a newLISP script from elsewhere, such as from inside a text editor, the current working directory and other settings may be different. It's a good idea, therefore, to use change-dir to establish the current working directory, if you're not sure:
Other environment variables can be accessed with env:
Again, running newLISP from a text editor rather than in an interactive terminal session will affect which environment variables are available and their values.
Once you have the correct working directory, use directory to list its contents:
and so on. Notice that it gives you the relative filenames, not the absolute pathnames. directory can list the contents of directories other than the current working directory too, if you supply a path. And in this extended form, you can use regular expressions to filter the contents:
Again, notice that the results are relative to the current working directory. It's often useful to store the path of the directory that you're listing, in case you have to use it later to build full pathnames. real-path returns the full pathname of a file or directory, either in the current working directory:
or specified by another relative pathname:
To find the containing directory of an item on disk, you can just remove the file name from the full pathname:
This won't always work, by the way, if the file name appears as a directory name as well earlier in the path. A simple solution is to do a regex search for f occurring only at the very end of the pathname, using the $ option:
To scan a section of your file system recursively, use a function that calls itself recursively. Here, just the full path name is printed:
See also Editing text files in folders and hierarchies.
You'll find some testing functions useful:
* file? does this file or directory exist?
* directory? is this pathname a directory or a file?
Remember the difference between relative and absolute pathnames:

### File information

You can get information about a file with file-info. This function asks the operating system about the file and returns the information in a series of numbers:
* 0 is the size
* 1 is the mode
* 2 is the device mode
* 3 is the user id
* 4 is the group id
* 5 the access time
* 6 the modification time
* 7 the status change time
To find out the size of files, for example, look at the first number returned by file-info. The following code lists the files in a directory and includes their sizes too.
Notice that we stored the name of the directory in dir. The directory function returns relative file names, but you must pass absolute pathname strings to file-info unless the string refers to a file that's in the current working directory.
You can use implicit addressing to select the item you want. So instead of (nth 0 (file-info item)), you can write (file-info item 0).

### MacOS X: resource forks

If you tried the previous script on MacOS X, you might notice that some files are 0 bytes in size. This might indicate the presence of the dual-fork system inherited from the days of the classic (ie old) Macintosh. Use the following version to access the resource forks of files. A good hunting ground for the increasingly rare resource fork is the fonts folder:

### File management

To manage files, you can use the following functions:
* rename-file renames a file or directory
* copy-file copies a file
* delete-file deletes a file
* make-dir makes a new directory
* remove-dir removes an empty directory
For example, to renumber all files in the current working directory so that the files sort by modification date, you could write something like this:
The (file-info item 6) extracts the modification time (item 6) of the result returned by file-info.
Always test scripts like this before you use them for real work! A misplaced punctuation character can wreak havoc.

## Reading and writing data

newLISP has a good selection of input and output functions.
An easy way of writing text to a file is append-file, which adds a string to the end of a file. The file is created if it doesn't exist. It's very useful for creating log files and files that you write to periodically:
and there's now a file on my desktop with the following contents:
You don't have to worry about opening and closing the file.
To load the contents of a file into a symbol in one gulp, use read-file.
The symbol contents stores the file's contents as a single string.
open returns a value which acts as a reference or 'handle' to a file. You'll probably want to use the file later, so store the reference in a symbol:
Use read-line to read a file one line at a time from a file handle. Each time you use read-line, the next line is stored in a buffer which you can access with the current-line function. The basic approach for reading a file is like this:
read-line discards the line feed at the end of each line. println adds one at the end of the text you supply. For more about argument handling and main-args, see STDIO.
For small to medium-size files, reading a source file line by line is much slower than loading the whole file into memory at once. For example, the source document for this book is about 6000 lines of text, or about 350KBytes. It's about 10 times faster to process the file using read-file and parse, like this:
than using read-line, like this:
The device function is a handy way of switching output between the console and a file:
Suppose that your script accepts a single argument, and you want to write the output to a file with the same name but with a .out suffix. Try this:
Now you can process the file's contents and output any information using println statements.
The load and save functions are used to load newLISP source code from a file, and to save source code into a file.
The read-line and write-line functions can be used to read and write lines to threads as well as files. See Reading and writing to threads.

### Standard input and output

To read from STDIO (standard input) and write to STDOUT (standard output), use read-line and println. For example, here's a simple filter that converts standard input to lower-case and outputs it to standard output:
And it can be run in the shell like this:
The following short script is a useful newLISP formatter submitted to the user forum by user Echeam:
which you can run from the command-line like this:

### Command-line arguments

To use newLISP programs on the command line, you can access the arguments with the main-args function. For example, if you create this file:
make it executable, and then run it in the shell, you'll see a list of the arguments that were supplied to the script when you run it:
main-args returns a list of the arguments that were passed to your program. The first two arguments, which you probably don't want to process, are the path of the newLISP program, and the pathname of the script being executed:
So you probably want to process the arguments starting with the one at index 2:
which is returned as a string. Or, to process all the arguments, starting at index 2, use a slice:
and the arguments are returned in a list of strings.
Often, you'll want to work through all the main arguments in your script: a convenient phrase for this is:
A slightly more readable equivalent is this, which works through the rest of the rest of the arguments:
Here's a short script that filters out unwanted Unicode characters from a text file, except for a few special ones that are allowed through:
For some more examples of argument processing, see Simple countdown timer.

# Working with pipes, threads, and processes
## Processes, pipes, threads, and system functions

The following functions allow you to interact with the operating system:
* ! runs a command in the operating system
* abort stops all spawned processes
* destroy kills a process
* exec runs a process and reads from or writes to it
* fork launches a newLISP child process thread (Unix)
* pipe creates a pipe for interprocess communication
* process launches a child process, remap standard I/O and standard error
* semaphore creates and controls semaphores
* share shares memory with other processes and threads
* spawn creates a new newLISP child process
* sync monitors and synchronizes spawned processes
* wait-pid waits for a child process to end
Because these commands interact with your operating system, you should refer to the documentation for platform-specific issues and restrictions.
! runs a system command and shows you the results in the console. The exec function does a similar job, but it waits for the operating system to finish, then returns the standard output as a list of strings, one for each line:
As usual, you'll have all the fun of quoting and double-quoting to get your commands through the shell.
With exec, your script will wait until the command finishes before your script continues.
You'll see the results only when the command finishes.
To interact with another process running alongside newLISP, rather than wait until a process finishes, use process. See Processes.

### Multitasking

All the newLISP expressions we've evaluated so far have run serially, one after the other, so one expression has to finish evaluating before newLISP can start on the next one. This is usually OK. But sometimes you want to start an expression and then move on to another while the first is still being evaluated. Or you might want to divide a large job into a number of smaller ones, perhaps taking advantage of any extra processors your computer has. newLISP does this type of multitasking using three functions: spawn for creating new processes to run in parallel, sync for monitoring and finishing them, and abort for stopping them before they've finished.
For the examples that follow, I'll use this short 'pulsing' function:
When you run this normally, you'll see 20 characters printed, one every interval milliseconds. The execution of this function blocks everything else, and you'll have to wait for all 20, or stop execution using Control-C.
To run this function in a parallel process alongside the current one, use the spawn function. Supply a symbol to hold the result of the expression, followed by the expression to be evaluated:
The number returned by the function is the process ID. You can now carry on using newLISP in the terminal while the pulsar continues to interrupt you. It's very irritating - dots keep appearing in the middle of your typing!
Start a few more:
To see how many processes are active (and haven't yet finished), use the sync function without arguments:
If you want to stop all the pulsar processes, use abort:
On MacOS X, try this more entertaining version that uses the resident spoken voice:
sync also lets you keep an eye on currently running processes. Supply a value in milliseconds; newLISP will wait for that time, and then check to see if the spawned processes have finished:
If the result is nil, then the processes haven't finished. If the result is true, then they've all finished. Now - and only now, after the sync function runs - the value of the return symbol r1 is set to be the value returned by the process. In the case of the pulsar, this will be the string "I've finished".
Notice that the process finished - or rather, it printed its concluding message - but the symbol r1 wasn't set until the sync function was executed and returned true. This is because sync returns true, and the return symbols have values, only if all the spawned processes have finished.
You could execute a loop if you wanted to wait for all the processes to complete:
which checks every second to see if the processes have completed.
As a bonus, many modern computers have more than one processor, and your scripts might be able to run faster if each processor can devote its energies to one task. newLISP leaves it to the operating system to schedule tasks and processors according to the hardware at its disposal.

### forked processes

There are some lower-level functions for manipulating processes. These aren't as convenient or as easy to use as the spawned process technique described in the previous section, but they provide a few additional features that you might find a use for some day.
You can use fork to evaluate an expression in another process. Once the process is launched, it won't return a value to the parent process, so you'll have to think about how to obtain its results. Here's a way of calculating prime numbers in a separate process and saving the output in a file:
Here, the new child process launched by fork knows how to find prime numbers, but, unlike the spawned processes, can't return information to its parent process to report how many it's found.
It is possible for processes to share information. The share function sets up a common area of memory, like a notice board, that all processes can read and write to. There's a simple example of share in a later chapter: see A simple IRC client.)
To control how processes access shared memory, newLISP provides a semaphore function.

## Reading and writing to threads

If you want forked threads to communicate with each other, you'll have to do a bit of plumbing first. Use pipe to set up communications channels, then arrange for one thread to listen to another. pipe returns a list containing read and write handles to a new inter-process communications pipe, and you can then use these as channels for the read-line and write-line functions to read and write to.
Notice how the "they've started" string appears before any primes are printed, even though that println expression occurs after the threads start.
The wait-pid function waits for the thread launched by fork to finish - of course you don't have to do this immediately.

## Communicating with other processes

To launch a new operating system process that runs alongside newLISP, use process. As with fork, you first want to set up some suitable plumbing, so that newLISP can both talk and listen to the process, which in the following example is the Unix calculator bc. The write-buffer function writes to the myout pipe, which is read by bc via bcin. bc's output is directed through bcout and read by newLISP using read-line.

# Working with XML
## Converting XML into lists

XML files are widely used these days, and you might have noticed that the highly organized tree structure of an XML file is similar to the nested list structures that we've met in newLISP. So wouldn't it be good if you could work with XML files as easily as you can work with lists?
You've met the two main XML-handling functions already. (See ref and ref-all.) The xml-parse and xml-type-tags functions are all you need to convert XML into a newLISP list. (xml-error is used for diagnosing errors.) xml-type-tags determines how the XML tags are processed by xml-parse, which does the actual processing of the XML file, turning it into a list.
To illustrate the use of these functions, we'll use the RSS newsfeed from the newLISP forum:
and store the retrieved XML in a file, to save hitting the server repeatedly:
The XML starts like this:
If you use xml-parse to parse the XML, but don't use xml-type-tags first, the output looks like this:
Although it looks a bit LISP-y already, you can see that the elements have been labelled as "ELEMENT", "TEXT". For now, we could do without all these labels, and that's basically the job that xml-type-tags does. It lets you determine the labels for four types of XML tag: TEXT, CDATA, COMMENTS, and ELEMENTS. We'll hide them altogether with four nils. We'll also use some options for xml-parse to tidy up the output even more.
This is now a useful newLISP list, albeit quite a complicated one, stored in a symbol called sxml. (This way of representing XML is called S-XML.)
If you're wondering what that 15 was doing in the xml-parse expression, it's just a way of controlling how much of the auxiliary XML information is translated: the options are as follows:
* 1 - suppress whitespace text tags
* 2 - suppress empty attribute lists
* 4 - suppress comment tags
* 8 - translate string tags into symbols
* 16 - add SXML (S-expression XML) attribute tags
You add them up to get the options code number - so 15 (+ 1 2 4 8) uses the first four of these options: suppress unwanted stuff, and translate strings tags to symbols. As a result of this, new symbols have been added to newLISP's symbol table:
These correspond to the string tags in the XML file, and they'll be useful almost immediately.

### Now what?

The story so far is basically this:
which has given us a list version of the news feed stored in the sxml symbol.
Because this list has a complicated nested structure, it's best to use ref and ref-all rather than find to look for things. ref finds the first occurrence of an expression in a list and returns the address:
These numbers are the address in the list of the first occurrence of the symbol item: (0 9 0) means start at item 0 of the whole list, then go to item 9 of that item, then go to item 0 of that one. (0-based indexing, of course!)
To find the higher-level or enclosing item, use chop to remove the last level of the address:
This now points to the level that contains the first item. It's like chopping the house number off an address, leaving the street name.
Now you can use this address with other expressions that accept a list of indices. The most convenient and compact form is probably the implicit address, which is just the name of the source list followed by the set of indices in a list:
That found the first occurrence of entry, and returned the enclosing portion of the SXML.
Another technique available to you is to treat sections of the list as association lists:
Here we've found the first item, as before, and then looked up the first occurrence of title using lookup.
Use ref-all to find all occurrences of a symbol in a list. It returns a list of addresses:
With a simple list traversal, you can quickly show all the titles in the file, at whatever level they may be:
Without the two rests in there, you would have seen this:
As you can see, there are many different ways to access the information in the SXML data. To produce a concise summary of the news in the XML file, one approach is to go through all the items, and extract the title and description entries. Because the description elements are a mass of escaped entities, we'll write a quick and dirty tidying routine as well:

### Changing SXML

You can use similar techniques to modify data in XML format. For example, suppose you keep the periodic table of elements in an XML file, and you wanted to change the data about elements' melting points, currently stored in degrees Kelvin, to degrees Celsius. The XML data looks like this:
When the table has been loaded into the symbol sxml, using (set 'sxml (xml-parse xml 15)) (where xml contains the XML source), we want to change each sublist that has the following form:
You can use the set-ref-all function to find and replace elements in one expression. First, here's a function to convert a temperature from Kelvin to Celsius:
Now the set-ref-all function can be called just once to find all references and modify them in place, so that every melting-point is converted to Celsius. The form is:
where the function is the method used to find list elements given the key.
Here the match function searches the SXML list using a wild-card construction (MELTING_POINT ( (UNITS "Kelvin") ) *) to find every occurrence. The replacement expression builds a replacement sublist from the matched expression stored in $0. After this has been evaluated, the SXML changes from this:
to this:
XML isn't always as easy to manipulate as this - there are attributes, CDATA sections, and so on.

## Outputting SXML to XML

If you want to go the other way and convert a newLISP list to XML, the following function suggests one possible approach. It recursively works through the list:
Which is - almost - where we started from!

## A simple practical example

The following example was originally set in the shipping department of a small business. I've changed the items to be pieces of fruit. The XML data file contains entries for all items sold, and the charge for each. We want to produce a report that lists how many at each price were sold, and the total value.
Here's an extract from the XML data:
This is the main function that defines and organizes the tasks:
Two functions are called: scan-file, which scans an XML file and stores the required information in a table, which is going to be some sort of newLISP list, and write-report, which scans this table and outputs a report.
The scan-file function receives a pathname, converts the file to SXML, finds all the charge items (using ref-all), and keeps a count of each value. We allow for the fact that some of the free items are marked variously as No Charge or no charge or nocharge:
The write-report function sorts and analyses the table, keeping running totals as it goes:
The report requires a bit more fiddling about than the scan-file function, particularly as the user wanted - for some reason - the 0 and no charge items to be kept separate.

# The debugger

This section takes a brief look at the built-in debugger.
The key function is trace. To start and stop the debugger, use true or nil:
Used without arguments, it returns true if the debugger is currently active.
The trace-highlight command lets you control the display of the expression that's currently being evaluated, and some of the prompts. I'm using a VT100-compatible terminal, so I can use weird escape sequences to set the colours. Type this in at the newLISP prompt:
But since I can never remember this, it's in my .init.lsp file, which loads when you start newLISP. If you can't use these sequences, you can use plain strings instead.
The other debugging function is debug. This is really just a short-cut for switching tracing on, running a function in the debugger, and then switching tracing off again. So, suppose we want to run a file called old-file-scanner.lsp, which contains the following code:
This scans a directory and subdirectories for files previously modified 5 or more years ago. (Notice that the file ends with expressions that will be evaluated immediately the file is loaded.) First, switch tracing on:
Then load and start debugging:
Or, instead of those two lines, type this one:
Either way, you'll see the first expression in the walk-tree function highlighted, awaiting evaluation.
Now you can press the s, n, or c keys (Step, Next, and Continue) to proceed through your functions: Step evaluates every expression, and steps into other functions as they are called; Next evaluates everything until it reaches the next expression at the same level; and Continue runs through without stopping again.
If you're clever, you can put a (trace true) expression just before the place where you want to start debugging. If possible, newLISP will stop just before that expression and show you the function it's just about to evaluate. In this case, you can start executing the script with a simple (load...) expression - don't use debug if you want to skip over the preliminary parts of the script. I think newLISP usually prefers to enter the debugger at the start of a function or function call - you probably won't be able to drop in to a function part way through. But you might be able to organize things so that you can do something similar.
Here is how to drop into the debugger halfway through a loop. Here's a small portion of code:
Load it from file:
Notice how the function f1 appeared - you don't get a chance to see anything in f2. At the debugger prompt, you can type in any newLISP expression, and evaluate any functions. Here I've typed i to see the current value of that symbol. newLISP is happy to let you change the values of some symbols, too. You should be able to change the loop variable, for example. But don't redefine any functions ... if you try to pull the rug from underneath newLISP's feet you'll probably succeed in making it fall over!
The source code displayed by the debugger doesn't include comments, so if you want to leave yourself helpful remarks - or inspiration - when looking at your code, use text strings rather than comments:

# The internet
## HTTP and networking

Most networking tasks are possible with newLISP's networking functions:
* base64-dec decode a string from BASE64 format
* base64-enc encode a string to BASE64 format
* delete-url delete a URL
* get-url read a file or page from the web
* net-accept accept a new incoming connection
* net-close close a socket connection
* net-connect connect to a remote host
* net-error return the last error
* net-eval evaluate expressions on multiple remote newLISP servers
* net-interface define the default network interface
* net-listen listen for connections to a local socket
* net-local local IP and port number for a connection
* net-lookup the name for an IP number
* net-peek number of characters ready to read
* net-peer remote IP and port for a net-connect
* net-ping send a ping packet (ICMP echo request) to one or more addresses
* net-receive read data on a socket connection
* net-receive-from read a UDP datagram on an open connection
* net-receive-udp read a UDP datagram on and closes connection
* net-select check a socket or list of sockets for status
* net-send send data on a socket connection
* net-send-to send a UDP datagram on an open connection
* net-send-udp send a UDP datagram and closes connection
* net-service translate a service name to a port number
* net-sessions return a list of currently open connections
* post-url post info to a URL address
* put-url upload a page to a URL address.
* xml-error return last XML parse error
* xml-parse parse an XML document
* xml-type-tags show or modify XML type tags
With these networking functions you can build all kinds of network-capable applications. With functions like net-eval you can start newLISP as a daemon on a remote computer and then use on a local computer to send newLISP code across the network for evaluation.

## Accessing web pages

Here's a very simple example using get-url. Given the URL of a web page, obtain the source and then use replace and its list-building ability to generate a list of all the JPEG images on that page:

## A simple HTML Form

The simplest search form is probably something like this.

## A simple IRC client

The following code implements a simple IRC (Internet Relay Chat) client, and it shows how the basic network functions can be used. The script logs in to the server using the given username, and joins the # newlisp channel. Then the script divides into two threads: the first thread displays any channel activity in a continuous loop, while the second thread waits for input at the console. The only communication between the two threads is through the shared connected flag.

# More examples

This section contains some simple examples of newLISP in action. You can find plenty of good newLISP code on the web and in the standard newLISP distribution.

## On your own terms

You might find that you don't like the names of some of the newLISP functions. You can use constant and global to assign another symbol to the function:
You can now use set! instead of setf. There's no speed penalty for doing this.
It's also possible to define your own alternatives to built-in functions. For example, earlier we defined a context and a default function that did the same job as println, but kept a count of how many characters were output. For this code to be evaluated rather than the built-in code, do the following.
First, define the function:
Keep the original newLISP version of println available by defining an alias for it:
Assign the println symbol to your Output function:
Now you can use println as usual:
And it appears to do the same job as the original function. But now you can also make use of the bonus features of the alternative println that you defined:
In case you've been counting carefully - the counter has been counting the length of the arguments supplied to the Output function. These include the parentheses, of course...

## Using a SQLite database

Sometimes it's easier to use existing software rather than write all the routines yourself, even though it can be fun designing something from the beginning. For example, you can save much time and effort by using an existing database engine such as SQLite instead of building custom data structures and database access functions. Here's how you might use the SQLite database engine with newLISP.
Suppose you have a set of data that you want to analyse. For example, I've found a list of information about the elements in the periodic table, stored as a simple space-delimited table:
(You can find the list in this file on GitHub..)
The columns here are Atomic Weight, Melting Point, Boiling Point, Density, Percentage in the earth's crust, Year of discovery, Group, and Ionization energy. (I used 0 to mean Not Applicable, which was not a very good choice, as it turns out).
To load newLISP's SQLite module, use the following line:
This loads in the newLISP source file which contains the SQLite interface. It also creates a new context called sql3, containing the functions and symbols for working with SQLite databases.
Next, we want to create a new database or open an existing one:
This creates a new SQLite database file called periodic_table, and opens it. If the file already exists, it will be opened ready for use. You don't have to refer to this database again, because newLISP's SQLite library routines maintain a current database in the sql3 context. If the open function fails, the most recent error stored in sql3:error is printed instead.
I've just created this database, so the next step is to create a table. First, though, I'll define a symbol containing a string of column names and the SQLite data types each one should use. I don't have to do this, but it probably has to be written down somewhere, so, instead of the back of an envelope, write it in a newLISP symbol:
Now I can make a function that creates the table:
It's easy because I've just created the column-def symbol in exactly the right format! This function uses the sql3:sql function to create a table called t1.
I want one more function: one that fills the SQLite table with the data stored in the list elements. It's not a pretty function, but it does the job, and it only has to be called once.
This function calls parse twice. The first parse breaks the data into lines. The second parse breaks up each of these lines into a list of fields. Then I can use format to enclose the value of each field in single quotes, remembering to change the strings to integers or floating-point numbers (using int and float) according to the column definitions.
It's now time to build the database:
- if the t1 table doesn't exist in a list of tables, the functions that create and fill it are called.

### Querying the data

The database is now ready for use. But first I'll write a simple utility function to make queries easier:
This function submits the supplied text, and either prints out the results (by mapping println over the results list) or displays the error message.
Here are a few sample queries.
Find all elements discovered before 1900 that make up more than 2% of the earth's crust, and display the results sorted by their discovery date:
When were the noble gases discovered (they are in group 18)?
What are the atomic weights of all elements whose symbols start with A?
It's elementary, my dear Watson! Perhaps the scientists out there can supply some examples of more scientifically interesting queries?
You can find MySQL and Postgres modules for newLISP on the net as well.

## Simple countdown timer

Next is a simple countdown timer that runs as a command-line utility. This example shows some techniques for accessing the command-line arguments in a script.
To start a countdown, you type the command (the name of the newLISP script) followed by a duration. The duration can be in seconds; minutes and seconds; hours, minutes, and seconds; or even days, hours, minutes, and seconds, separated by colons. It can also be any newLISP expression.
or:
or:
or:
Alternatively, you can supply a newLISP expression instead of a numerical duration. This might be a simple calculation, such as the number of seconds in π minutes:
or, more usefully, a countdown to a specific moment in time, which you supply by subtracting the time now from the time of the target:
- in this example we've specified Christmas Day using date-value, which returns the number of seconds since 1970 for specified dates and times.
The evaluation of expressions is done by eval-string, and here it's applied to the input text if it starts with "(" - generally a clue that there's a newLISP expression around! Otherwise the input is assumed to be colon-delimited, and is split up by parse and converted into seconds.
The information is taken from the arguments given on the command line, and extracted using main-args, which is a list of the arguments that were used when the program was run:
This fetches argument 2; argument 0 is the name of the newLISP program, argument 1 is the name of the script, so argument 2 is the first string following the countdown command.
Save this file as countdown, and make it executable.

## Editing text files in folders and hierarchies

Here's a simple function that updates some text date stamps in every file in a folder, by looking for enclosing tags and changing the text between them. For example, you might have a pair of tags holding the date the file was last edited, such as <last-edited> and </last-edited>.
which can be called like this:
The replace-string-in-files function accepts a folder name. The first task is to extract a list of suitable files - we're using directory with the regular expression {^[^.]} to exclude all files that start with a dot. Then, for each file, the contents are loaded into a symbol, the replace function replaces text enclosed in the specified strings, and finally the modified text is saved back to disk. To call the function, specify the start and end tags, followed by the text and the folder name. In this example we're just using a simple ISO date stamp provided by date and date-value.

### Recursive version

Suppose we now wanted to make this work for folders within folders within folders, ie to traverse a hierarchy of files, changing every file on the way down. To do this, re-factor the replace-string function so that it works on a passed pathname. Then write a recursive function to look for folders within folders, and generate all the required pathnames, passing each one to the replace-string function. This re-factoring might be a good thing to do anyway: it makes the first function simpler, for one thing.
Next for that recursive tree-walking function. This looks at each normal entry in a folder/directory, and tests to see if it's a directory (using directory?). If it is, the replace-in-tree function calls itself and starts again at the new location. If it isn't, the pathname of the file is passed to the replace-string-in-file function.
To change a tree-full of files, call the function like this:
It's important to test these things in a scratch area first; a small mistake in your code could make a big impact on your data. Caveat newLISPer!

## Talking to other applications (MacOS X example)

newLISP provides a good environment for gluing together features found in application programs that have their own scripting languages. It's fast and small enough to keep out of the way of the other components of a scripted solution, and it's good for processing information as it passes through your workflow.
Here is an example of how you can use a newLISP script to send non-newLISP scripting commands to applications. The task is to construct a circle in Adobe Illustrator, given three points on the circumference.
The solution is in three parts. First, we obtain the coordinates of the selection from the application. Next we calculate the radius and centre point of the circle that passes through these points. Finally, we can draw the circle. The first and final parts use AppleScript, which is run using the osascript command, because Adobe Illustrator doesn't understand any other scripting language (on Windows, you use Visual Basic rather than AppleScript).
The calculations and general interfacing are carried out using newLISP. This can often be a better solution than using native AppleScript, because newLISP offers many powerful string and mathematical functions that can't be found in the default AppleScript system. For example, if I want to use trigonometry I would have to find and install an extra component - AppleScript doesn't provide any trig functions at all.
The newLISP script can sit in the Scripts menu on the menu bar; put it into the Library > Scripts > Applications > Adobe Illustrator folder, which accepts both text files and AppleScripts). It's then ready to be selected while you're working in Illustrator. To use it, just select a path with at least three points, then run the script. The first three points define the location for the new circle.
There's hardly any error handling in this script! More should definitely be added to the first stage (because the selection might not be suitable for subsequent processing).

# Graphical interfaces
## Introduction

With newLISP you can easily build graphical interfaces for your applications. This introductory document is long enough already, so I won't describe the newLISP-GS feature set in any detail. But there's room for a short example to give you a taste of how it works.
The basic components of newLISP-GS are containers, widgets, events, and tags. Your application consists of containers, which hold widgets and other containers. By giving everything a tag (a symbol), you can control them easily. And when the user of the application clicks, presses, and slides things, events are sent back to newLISP-GS, and you write code to handle each event.

## A simple application

To introduce the basic ideas, this chapter shows how easy it is to build a simple application, a colour mixer:
You can move the sliders around to change the colour of the central area of the window. The colour components (numbers between 0 and 1 for red, green, and blue), are shown in a text string at the bottom of the window.
In newLISP-GS the contents of a container are arranged depending on the type of layout manager you choose - at present you can have flow, grid, or border layouts.
The following diagram shows the structure of the application's interface. The primary container, which in this case is a frame called 'Mixer', is filled with other containers and widgets. The top area of the window, containing the sliders, consists of a panel called 'SliderPanel', which in turn is filled with three panels, one for each slider, called 'RedPanel', 'GreenPanel', and 'BluePanel'. Below, the middle area holds a canvas called 'Swatch' to show the colour, and at the bottom area there's a text label, called 'Value', displaying the RGB values as text. Each area is laid out using a different layout manager.
There's just a single handler required. This is assigned to the sliders, and it's triggered whenever a slider is moved.
The first step is to load the newLISP-GS module:
This provides all the objects and functions required, in a context called gs.
The graphics system is initialized with a single function:
The various parts of the interface can be added one by one. First I define the main window, and choose the border layout. Border layouts let you place each component into one of five zones, labelled "north", "west", "center", "east" and "south".
The top panel to hold the sliders can now be added. I want the sliders to be stacked vertically, so I'll use a grid layout of 3 rows and 1 column:
Each of the three colour panels, with its companion labels and slider, is defined. The sliders are assigned the slider-handler function. I can write that when I've finished defining the interface.
The gs:add-to function adds components to a container, using the layout that's been assigned to it. If no layout has been assigned, the flow layout, a simple sequential layout, is used. Specify the target container first, then give the components to be added. So the objects tagged with 'Red, 'RedSlider, and 'RedSliderStatus are added one by one to the 'RedPanel container. When the three panels have been done, they can be added to the SliderPanel:
You can draw all kinds of graphics on a canvas, although for this application I'm just going to use a canvas as a swatch, a single area of colour:
Now the three main components - the slider panel, the colour swatch, and the label for the value - can be added to the main frame. Because I assigned the border layout to the frame, I can place each component using directions:
We haven't used the east and west areas.
By default, frames and windows start life as invisible, so now is a good time to make our main frame visible:
That completes the application's structure. Now some initialization is required, so that something sensible appears when the application launches:
Finally, I mustn't forget to write the handler code for the sliders. The handler is passed the id of the object which generated the event, and the slider's value. The code converts the value, an integer less than 100, to a number between 0 and 1. Then the set-color function can be used to set the colour of the canvas to show the new mixture.
Only one more line is necessary and we've finished. The gs:listen function listens for events and dispatches them to the handlers. It runs continuously, so you don't have to do anything else.
This tiny little application has barely scratched the surface of newLISP-GS, so take a look at the documentation and have a go!