Source: https://code.kx.com/q4m3/11_IO/
Timestamp: 2019-04-24 08:24:33+00:00

Document:
I/O in q is one of the most powerful and succinct features of the language. The names and behavior of the functions are idiosyncratic but the economy of expression is unrivaled.
I/O is realized via handles, which are symbolic names of resources such as files or machines on a network. One-and-done operations can be performed directly on the symbolic handle – e.g., you can read a file into memory in a single operation. For continuing operations, you open the symbolic handle to obtain an open handle. The open handle is a function that is applied to perform operations. When you have completed the desired operations, you close the open handle to free any allocated resources.
In q, files come in two flavors: text and binary. Routines to process text data have ‘0’ in their names, whereas routines to process binary data have ‘1’. A text file is considered to be a list of strings – i.e., a list of char lists – and a binary file is a list of byte lists. While all text files can also be processed as binary data, not all binary data represents text. As mentioned above, file operations use handles.
where the bracketed expression represents an optional path and name is a file or directory name. The combination should be recognized as valid by the underlying operating system.
Some q operations require that you append a trailing slash / to indicate that you mean a directory. We will point these out.
It is generally easier to work with paths and names as strings so that blanks and other special characters can be handled easily. While `$ converts a string to a symbol, it can be awkward to include the leading : required in the symbolic handle. The operator hsym, which inserts a leading colon into a symbol, serves this purpose.
Note that q always represents separators in paths by the forward slash /, even when running on Windows. If you run q on Windows, you can type either / or \ but q will always display / in its response.
To make life easier when you are generating paths dynamically, hsym is idempotent, meaning that it will accept its own output and pass it through.
The first one-and-done operation that works directly on a symbolic file handle is hcount, which returns a long representing the size of the file in bytes as reported by the OS.
The next one-and-done is hdel, which instructs the OS to remove the file specified by its symbolic handle operand.
The return value of the symbolic file handle itself indicates that the deletion was successful. It should not be confused with an error message, which starts with a tick rather than a backtick.
You will get an error message if the file does not exist or if the delete cannot be performed.
You will not be prompted for confirmation. Back up any files that are important.
Every q entity can be serialized and persisted to storage. Unlike traditional languages, where you must instantiate serializers and writers, things are simple and direct in q. This is because q data is self-describing, so that its internal representation can be written out as a sequence of bytes and then read directly back into memory. This is as close to the Star Trek transporter as we are likely to get.
The magic is done by (an overload of) the dyadic set, whose left operand is a file handle and right operand is the entity to be written. The result is the symbolic handle of the written file. The file is automatically closed once the write is complete.
The behavior of set is to create the file if it does not exist and overwrite it if it does. It will also create the directory path if it does not exist.
A serialized q data file can be read using (an overload of) the monadic get, whose argument is a symbolic file handle and whose result is the q entity contained in the data file.
An equivalent way to read a data file is with (an overload of) value.
As with traditional languages, for continuing operations on a q data file, you open the file, perform the operation(s) and then close it. Unlike traditional languages, opening a symbolic handle returns a function, called an open handle, that is used to perform operations.
As mentioned previously, q files come in two flavors, binary and text. Serialized q data persisted with set is written in binary form with a header at the beginning of the file. You can read it as raw binary data to inspect its internals.
Open a data file handle with hopen, whose result is a function called the open handle. This function should be stored in a variable, traditionally h, which is functionally applied to data to write it to the file. We will explain the result of applying the open handle shortly. We begin with a file containing serialized q data and show how to append to it.
Always apply hclose to the open handle to close it and flush any data that might be buffered. Failure to do so may cause your program to run out of file handles unnecessarily.
We verify that the appends have been made.
We can also create a new file and write raw binary data to it.
Now, what is the deal with the 3i return value of applying the open handle?
In fact, the return value is the value of the open handle itself.
Surely, you say, we can’t use an int as a function to write data. But you would be wrong.
Apparently q assigns an int to each open file and keeps track of which int values are valid handles. This accounts for the cryptic error message when you attempt to use variables with simple list notation.
Apply read1 on a file handle to read any file into q as a list of bytes. For example, we can read the previously serialized value L as bytes.
This shows the internal representation of the serialized q entity. How cool is that?
If you want to write raw binary data, as opposed to the internal representation of a q entity containing the data, use the infelicitously named 1:. It takes a symbolic file handle as its left argument and a simple byte list as its right argument. Bytes in the right operand are essentially streamed to the file.
Fundamentalists can use dot amend in place of set to serialize q entities to files. To write the file, or overwrite an existing file, use assign :.
To append to an existing file use ,.
We have already seen that it is easy to write and read tables to/from persistent storage.
The save and load functions make this even easier.
In its simplest form, save serializes a table in a global variable to a binary file having the same name as the variable. It overwrites an existing file.
This is equivalent to using set above with the table name as file name.
As you might expect, load is the inverse of save meaning that it reads a serialized table from a file into a variable with the same name as the file. It creates the variable in the workspace or overwrites it if it already exists.
You can also use save to write a table to a text file. You determine the format of the text with the file extension in the file handle.
All the following versions of save can also be performed with the more general 0: – see §11.5.
Save the table with .txt extension to obtain tab-delimited records. There is no corresponding load but you can parse the text file – see §11.5.1.
Save the table with .csv extension to obtain comma-separated values. There is no corresponding load but you can parse the CSV file – see §11.5.2.
Save the table with .xml extension to obtain XML records. There is no direct way to read XML into q although libraries have been contributed – see code.kx.com.
Save the table with .xls extension obtain an Excel spreadsheet. This file can be loaded by Excel work-alikes.
We have already seen how to persist a table to a file using set. There are no restrictions on the types of columns in the table or the file name in this scenario.
This creates a single file, as the OS verifies.
For larger tables that may not fit into memory on all machines, you can ask q to serialize each column of the table to its own file in a specified directory. A table persisted in this form is called a splayed table. The advantage is that when querying a splayed table, only the columns referred to in the query will be loaded into memory. This is a substantial memory win for a table having many columns.
It is worthwhile looking up the origin of the English word “splay”. Also, please don’t spay your tables.
To splay a table, use set and specify a directory as the target location indicated by a trailing slash / in the left operand.
List the directory in the OS and you will see a directory tsplay that contains three files, one file for each column in the original table, as well as a hidden .d file.
drwxr-xr-x 5 jeffry wheel 170 Mar 6 08:36 .
drwxr-xr-x 9 jeffry wheel 306 Mar 6 08:36 ..
Nearly all the metadata regarding the splayed table can be read from the file system – i.e., the name of table from directory and names of the columns from the files. The one missing bit is the order of the columns, which is stored as a serialized list in the hidden .d file.
There are restrictions on tables that can be splayed.
All columns must be simple or compound lists. The latter means a list of simple lists of uniform type. An arbitrary general list column cannot be splayed.
Symbol columns must be enumerated.
The convention for enumerating symbols in splayed tables is to enumerate all symbol columns in all tables over the domain sym and store the resulting sym list in the root directory – i.e., one level above the directory holding the splayed table. You can do this manually but practically no one does.
Normally folks use one of the .Q utilities, in spite of the official Kx admonition not to use them. For example, here we use .Q.en.
Only unofficially documented, .Q.en prepares a qualified table for splaying by enumerating all its symbol columns. The first argument is the symbolic file handle of the root directory for the persistent residence of the enumeration domain sym (no choice in the name). The second argument is a table. See §14.5.2 for more detail on its behavior.
.Q is now documented at code.kx.com. Ed.
We have seen that q views a record in a binary data file as a list of bytes. Similarly, a record in a text file is viewed as a list of char – i.e., a string. Thus reading a text file results in a list of strings and you pass a list of strings to write to a text file.
You can see the underlying binary values of the text by using read1 or casting the result of read0 to char.
Or you can read the data as binary and cast the result to char. Observe that the data is a simple list of char so the newline character does not cause line breaks in the console display.
To write string as text, use the (infelicitously named) dyadic 0:, which takes a file handle in the left operand and a list of strings in the right operand. It creates the directory path if necessary and overwrites the file if it already exists.
Just as with a binary data file, a symbolic text file handle can be opened with hopen. The result is again an int that is conventionally stored in the variable h and is used with function application syntax to write data. The difference is that instead of using plain h to write binary data, you use neg[h] to write strings as text. Seriously.
Observe that you apply hclose to h, not to neg[h].
If the file already exists, opening with hopen and applying the open handle will append rather than overwrite.
We saw the built-in functions for saving tables as text files in §11.2. When you need to control the filename, you can write the table yourself with 0:, but then you must prepare the table columns as formatted text. A separate overload of 0: is available for this purpose. A confusing naming convention, to say the least.
In this use, 0: has as left operand a char delimiter and as right operand a table or list of columns. Observe the use of the pre-defined constant csv, which is simply ",".
In the last snippet we applied 0: with two different meanings: to prepare and then write text. We hope you’ve grown fond of this name, since §11.5 will introduce yet another version of 0: for parsing text records.
Dyadic forms of 0: and 1: parse individual fields according to data type from text or binary records. Field parsing is based on the following field types.
The column labeled ‘0’ contains the (upper case) field type char for text data. The (lower case) char in column ‘1’ is for binary data. The column labeled ‘Width(1)’ contains the number of bytes that will be parsed for a binary read. The column labeled ‘Format(0)’ displays the format(s) that are accepted in a text read.
The parsed records are returned in column form rather than row form to make it easy to associate a list of symbol names with ! and then flip into a table.
The left operand is a nested list containing two items: Lt is a simple list of char containing one letter per field; Lw is a simple list of int containing one integer width per field. The sum of the field widths in Lw should equal the width of the record. The result of the function is a list of lists, one list arising from each field.
This reads a text file containing fixed length records of width 39. The first field is a long occupying 4 positions; the second field is a float occupying 8 positions; the third field consists of a symbol occupying 10 positions; the fourth slot of 6 positions is ignored; the fifth field is a date occupying 10 positions.
You might think that the widths are superfluous, but they are not. The actual data width can be narrower than the normal size due to small values, as in our case of the long field. Or you may need to specify a width larger than that required by the corresponding data type due to whitespace in the fields, as in the case of our float field.
Observe how easy it is to make a table from the result.
Also note that it is possible to parse a list of strings using the same format, since they represent text records in memory.
where hfile is a symbolic file handle, i is the offset into the file to begin reading and n is the number of bytes to read. This is useful for sampling a file or for large files that cannot be read into memory in a single gulp.
A read operation should begin and end on record boundaries or you will get meaningless results.
The left operand is a list comprising two lists. Lt is a simple list of char containing one type letter per corresponding field. D is either a char representing the delimiting character or an enlisted char.
Specify D as a delimiter char when the first record of the file does not contain column names. In this case, the result of the parse is a list of column lists, each of which contains items of type specified by Lt. The simplest form of the right operand f is a symbolic file handle.
Parsing with a delimiter char "," results in a list of column lists. As with parsing fixed format recodes, it is easy to make the result into a table.
Specify D as an enlisted char when the first record contains a separated list of names. Subsequent records are read as data specified by the types in Lt. The result is a table in which the column names are taken from the first record.
Reading with an enlisted "," delimiter results in a table.
The operator 0: can also be used to process text representing key-value pairs. In this situation, the left operand is a three-character string Pf that specifies the pair format. The first char of Pf can be "S" to indicate the key is a string or "I" to indicate the key is an integer. The second char indicates the key-value separator. The third char indicates the pair delimiter.
The following examples illustrate various combinations in Pf.
Again it is easy to make the result into a table.
The ease with which a q process can communicate with another q process residing on the network is one of the most impressive features of q. We shall cover all the basics of interprocess communication (IPC) so that you can follow the section on callbacks in Chapter 1 – Q Shock and Awe.
We shall use the following terminology. The process that initiates the communication is called the client, while the process receiving and processing requests is the server. The server process can be on the same machine, the same network, a different network or on the Internet, so long as it is accessible. The communication can be synchronous (wait for a result to be returned) or asynchronous (don’t wait and no result returned).
The only way to learn IPC is to do it, and the easiest way to do this is to set up two processes on the same machine. We recommend you use the machine running your q sessions for this tutorial, provided it will allow a port to be opened. In what follows, we shall assume that a server q process has been started on a machine with an open port.
The client process is a separate q process running on the same machine.
Here the bracketed expression represents an optional server machine identifier and port is a port number. An omitted server specification, or one of the form localhost, refers to the machine on which the originating q session lives. The following both refer to port 5042 on the same machine as the q session in which they are entered.
You can refer to a machine on the network by name. For example, on the author’s laptop the following is equivalent to the two previous network handles.
You can use the IP address of a machine.
Finally, you can also use a URL.
As with a file handle, apply hopen to a communication handle to obtain an open connection handle that is used as a function. As before, the value is an int that is traditionally stored in the variable h. Also as with file I/O, the behavior of this function differs between using the original positive handle or its negation.
Let’s see how this works with our two sessions. (You did start them, didn’t you?). Remember, the session that opened port 5042 is the server; the other session is the client. In the client session, open a handle to the server and store it in h, then apply h to the string as shown. Finally close the connection handle.
Whitespace between h and the quoted string is optional, as this is simply function juxtaposition. We include it for readability.
We have seen that when you open a connection to a q process, you have the full capability of that process available remotely. Apply the connection handle to any q expression in a string and it will be evaluated on the server. As you contemplate the IPC Zen, a dark cloud passes over your tranquility. You realize that, by default, the server is wide open.
Allowing quoted q strings to be executed on a server makes the server susceptible to all manner of breaches. Good practice does not permit this on a production server. You can mitigate this by having your server process accept only requests whose first item is a symbol (see below), which you should verify is the name of a function you have decided to expose.
The symbolic name of a map on the server.
The remaining items arg1, arg2, … are optional values sent along to the server for the evaluation. These are arguments when f is a function, indices when it is a list, or keys when it is a dictionary.
Application of the connection handle to such a list sends the list to the server where it is evaluated. Any result is sent back to the client, where it is presented as the result of the connection handle application. By simply applying the naked handle, this sequence of steps is synchronous, meaning that execution of the q session on the client blocks until the result of the server evaluation is returned.
Our examples will cover the case when f is of function type since that is most common. We first consider the first case when f is a map on the client side. In this situation the function (list, dictionary, etc.) is actually transported to the server along with the supplied arguments, where it is applied.
Allowing a function to be sent to the server for remote execution is as dangerous as sending quoted q strings. The function can access resources on the server and instigate an attack. Good practice does not permit this in production environments.
The remaining format for remote execution can be made safe for production environments. The function to be executed remotely must already be defined on the server and you pass its name and arguments via the connection handle.
Now “call” the function f remotely from the client.
The difference from SQL stored procedures is that the remote procedure can be any q function on the server, making the full power of q available remotely.
The IPC in the previous sections was synchronous, meaning that upon application of the connection handle, the client process blocks, waiting for a result from the server before proceeding. The value returned from the server becomes the return value of the open handle application.
Under the covers, IPC is implemented as messages passed over an open connection between q processes. When the positive open handle is applied to an argument, the message passing is synchronous, meaning that the following steps occur in sequence.
The client sends a message containing the argument(s) of the handle application to the server and waits for a return message.
The server receives the message, interprets it as the appropriate function application and obtains the result.
The server sends a message containing the result back to the client.
The client receives the result and resumes execution from the point it left off.
When a client sends multiple messages to a server in synchronous message passing, the next message is not sent until the result of the previous message is received. Consequently the messages always arrive at the server in the order in which they are sent. Also, the results from the server arrive back at the client in the order in which the original messages were sent.
It is also possible to perform asynchronous IPC in q. In this case the message is sent to the server and execution on the client continues immediately. In particular, there is no return value from the server. This is useful to initiate a task on the server when you don’t care about the result. For example, you could initiate a long running operation, or you could send a message that the server will route to other processes.
Use the negation of the open connection handle to send an asynchronous message to the server. Let’s define an instrumented function on the server to demonstrate what is happening.
You will observe 25 displayed on the server console. Also, the client session returns immediately with no return value. The expression on the console actually has a nil value :: that is suppressed by the console display.
When sending asynchronous messages, always send an empty “chaser” message immediately before applying hclose to the open handle. If you do not do this, buffered messages may not be sent when the connection is closed.
In order to convince ourselves that the client actually does return immediately without waiting for a return from the server, we wrap the client expression in a function. Observe that the client continues with the next statement.
Because a q session is single threaded by default, the server will process messages in the order in which they are received. However, in asynchronous messaging there is no guarantee that the messages arrive at the server in the order in which they are sent. It can be difficult to observe indeterminancy in simple examples, but you must assume that it will occur in practice.
Assuming that you have passed the server either a function from the client side or the name of a function on the server side, the appropriate function is evaluated on the server. During evaluation, the communication handle of the remote process is available in the system variable .z.w ( “who” called). For an asynchronous call, this can be used to send messages back to the server during the function application on the server.
Both the client and the server have connection handles when a connection between them is opened. However, these handles are assigned independently and their int values are not equal in general.
Here is a simple example showing how to use .z.w to send a message back to the client. On the server, we define a function that displays its received parameter and then asynchronously calls mycallback with the passed argument incremented.
On the client we define mycallback to display its parameter on the console. Then we make an asynchronous call to the function f on the server with an argument of 42.
The result is that "Received 42" is displayed on the server console and "Returned 43" is displayed on the client console. Congratulations! We have just invented callbacks in q.
When performing asynchronous messaging, always use neg[.z.w] to ensure that all messages are asynchronous. Otherwise you will get a deadlock as each process waits for the other.
You can override the default behavior of message processing in q by assigning your own handler(s) to the appropriate system variables. Assign your function to the variable .z.pg to trap and process synchronous messages and to .z.ps for asynchronous messages. The names end in ‘g’ and ‘s’ because synchronous processing has "get" semantics and asynchronous processing has "set" semantics.
In the following we set the asynchronous handler to a trivial function, essentially ignoring asynchronous calls.
On the client send an asynchronous message.
This results in "ignore" being displayed on the server console.
Now we set the synchronous handler to a function that only accepts “safe” remote calls by function name. It then performs a protected evaluation on the function with the arguments passed, thus ensuring that a failed application does not hang the server.
Now send synchronous messages from the client.
You can also specify handlers to be called upon connection open and close by assigning functions to the system variables .z.po and .z.pc, respectively. The connection handle of the sending process is passed as the lone argument to the functions assigned to .z.po and to .z.pc.
Here is a simple example that tracks connections and allows client processes to register callbacks with the server. Start a fresh q session on the server and open port 5042. Create a keyed table called Registry and define a function that can be invoked remotely to register a callback. Attach a handler to .z.po that initializes a dummy entry in Registry for the connection being opened and attach a handler to .z.pc to remove the record when a connection is closed.
Start a fresh q session on the client and connect to the server.
We check that an item has been entered into Registry on the server.
Next we register the name of a callback function from the client. Note the asynchronous message.
Again we check Registry on the server and observe that our callback name has indeed been registered.
Finally, we close the connection on the client.
And observe that the client has been automatically unregistered.
In this section, we demonstrate how to execute q-sql queries against a remote server. First, we splay a table to stand for a time-series database. We use the mktrades script that we created in §9.3.1 to create a trades table with 1,000,000 rows and then splay it to disk.
Now start a fresh server process (the server), open a port, say 5042, and map the splayed trade table into memory. Check that the mapping succeeded by running a query.
Leave the server process running and start another fresh process (the client), open a connection to the server and send the same query to the server for remote execution.
q)h "select from trade where dt=2015.01.01,sym=`ibm"
Now on the client we (synchronously) call the stored procedure by name with appropriate arguments.
In an actual application you would validate the input parameters and wrap the core evaluation in protected evaluation to trap unanticipated errors. You would also want to implement an entitlements system based on LDAP.
You should see 42 in the browser page display.
You can trap HTTP GET and POST traffic by assigning functions to the system variables .z.ph and .z.pp respectively. The default handler for .z.ph is to evaluate the content of the first item of the passed argument.
There is no default handler for .z.pp.
Here is a simple example that duplicates the default GET processing and shows the two items of its list argument. Define the following handler on the server process opened previously. It displays the two items of the input list then executes the first after removing the leading ? and then returns the result as a string.
Now enter the following from a browser on the same machine.
User-Agent      | "Mozilla/5.0 (Macintosh; Intel Mac OS X 10\_9\_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.89 Safari/537.36"
Accept-Encoding | "gzip, deflate, sdch"
And the browser page displays “42”.
WebSockets is a network protocol that upgrades an initial HTTP handshake into a TCP/IP socket connection. It was initially used to enhance communication capability between browsers and web servers but it can be used for general client-server applications. Once the WebSocket connection is established, either the client or server can message the other; in particular, this provides the capability for the server to push data to the client.
As of this writing (Sep 2015) q implements only asynchronous messaging in WebSockets.
In this section we show the basic mechanism for establishing a WebSocket connection between a browser and a q process acting as the server. We use Chrome for the examples but recent versions of Internet Explorer are now WebSockets-capable and should work similarly.
We begin with an extremely simple HTML page with a button that, when clicked, displays the answer to life, the universe and everything. Save the following as a text file sample0.html in a location accessible to your browser.
You should see a page with a single button labeled “get the answer”. Click the button and you will see the answer in a very large font.
Now we enhance this basic page to connect to a q process via WebSockets and retrieve the answer from q. Save the following script as sample1.html. We explain it below.
For simplicity in the example, we have placed a copy of c.js in the pages directory. You should modify this to reflect its location in your installation.
This script first declares the script c.js, which is required for using q WebSockets.
ws to hold a WebSocket object.
The function connect() is where the WebSocket action happens.
It first tests to see if WebSocket is in the window, meaning that the browser supports WebSockets. If so, it makes the connection to the server; otherwise it displays an error alert.
The first step in the connection is to create a WebSocket object by connecting to the specified server URL, and storing the result in ws.
Then set the binaryType field in ws to the value needed by the q sockets code.
Now we assign handlers for the main WebSockets events.
The close handler is empty.
The message handler deserializes the data field of the parameter e and applies the sayN function to display the result on the page.
The error handler displays an alert page with the error message.
The sayN function locates the answer field on the page and places the text of its argument there. Finally, the script defines a simple HTML element answer.
In contrast, the server side q code is blissfully short. Start a fresh q session, open port 5042 and set the WebSockets handler .z.ws to a function that will be invoked to handle WebSockets messages.
The handler first deserializes its parameter and displays it to the console for debugging, at which point we have no further use for it in this example. Then it serializes the answer to the question asked by the browser and asynchronously sends it back to the browser. That’s all there is to it!
and you will see the answer displayed on the page. At this point you are equipped to follow §1.19 in Q Shock and Awe.
In ordinary Web applications, the browser initiates interaction with the server. It sends a request to a specific URL on the server and the server replies with the requested page or data. Each such interaction is self-contained and is synchronous in that the browser waits for the server response.
In WebSockets the browser initiates the connection, but once the WebSocket request for protocol upgrade is successful, the browser – i.e., client – and the server are on equal footing. Either side can send messages. Moreover, in the current q implementation of WebSockets all interaction is asynchronous. Given that most current browsers and the default q session are both single-threaded, you don’t have to worry about races and deadlocks but you do have to set up callbacks.
In this section we demonstrate how the q server can push data to the browser, beginning with the browser script. Actually this script is a simplification of sample1.html in that we remove the initial call to the server upon open; everything else remains the same. The key point is that the onmessage handler will be called every time data is received, resulting in the data being displayed on the screen. Save the following as sample2.html.
And now for the q side. You can enter the following in the console of a fresh q session; or you can save it as a script and load it with \l.
Here is what’s happening in the q code.
First we open the port and initialize the answer variable.
Then we set the connection open handler to store the client .z.w value of its parameter into the global requestor and start the system timer firing every 1000 milliseconds. Note that this only happens after the browser initiates a connection.
Finally, we set the timer handler to send an asynchronous message containing the serialized value of answer and then increment answer.
and you will see the answer ticking every second on the page.

References: §11
 §11
 §11
 §14
 §11
 §11
 §9
 §1