The AWK Programming Language: An Introduction
By: Dane Bulat
Tags: Awk, Programming Languages, Linux, Computing, Linux Tutorial

Running AWK Programs

An AWK program is data-driven, which means that it performs a set of actions on the textual data it receives. There are two methods for supplying data to a program: Specifying a list of filenames, and piping output from another command. These approaches can be observed with some pseudo code:

# Providing input files

awk 'program' input-file1 input-file2 # Piping the output of a command

command | awk 'program'

In addition, the program portion can be either:

A one-shot throw away program that is quick to type and specified on the command line.

A script defining a sophisticated program that performs more complex data processing.

The best approach to take will depend on your program’s overall complexity and whether you want to re-use it in the future. The following pseudo code illustrates how to provide an AWK program:

# Command line

awk 'program' input-file1 input-file2 ... # Program file

awk -f program-file input-file1 input-file2 ...

Notice that a program specified directly on the command line is surrounded in single quotes. Alternatively, you can provide a program file with the -f flag.

It is also a good idea to append the .awk file extension to your scripts for a couple of reasons. It enables editors to render syntax highlighting for the AWK programming language correctly, and it immediately conveys the nature of the file when it's listed in a file manager or terminal.

Running Throw-Away Programs

Here are some quick one-shot example programs invoked directly on the command line. Feel free to run each one in your terminal, and don’t worry if the syntax makes no sense at this point:

# Output total number of fonts in /usr/share/fonts

$ fc-list | awk 'END { print "Total fonts:", NR }' # Output all (hidden) files starting with ".bash"

$ ls -la | awk '$NF ~ /^.bash/ { print $NF }' # Output all filenames that contain more than 10 characters

# (minus file extension)

$ ls | awk '{ i=index($0,"."); i>0?s=substr($0,1,i-1):s=$0;

if(length(s)>10) print }'

Throw-away programs can also process text files:

# Create a text file listing system fonts

$ fc-list > font-list # Provide the text file and output total number of fonts

$ awk 'END { print "Total fonts:", NR}' font-list

Programs specified on the command line can be more than one line, but you will need to use the backslash character to separate each one:

$ ls -l | awk 'BEGIN { print "About to count files..." } \

{ ++count } \

END { print "Total files: ", count }'

These types of programs are great for getting information quickly and experimentation. However, writing programs in text files means there is no limit to the complexity or length of your program.

Running AWK Scripts

We will use a script provided in this article’s corresponding GitHub repository to discuss the concepts in this section. You can download the GitHub repository containing this article’s code examples using git on the command line:

The program discussed in this section is located at samples/filemod.awk . The script renames files in the current working directory by converting all the characters to either lowercase or uppercase. When no command-line options are specified, all filenames are converted to lowercase:

# Enter samples directory

$ cd awk-samples/samples # Add executable permissions to the scripts

$ chmod +x *.awk # Invoke script

$ ls -la | ./filemod.awk

After piping ls -la to ./filemode.awk you should see some formatted output informing you of any modifications to the filenames:

Running the samples/filemod.awk program.

Let’s discuss the commands that were necessary to run the script:

chmod +x *.awk

In order to use the ./program.awk shorthand syntax, the scripts must have executable permissions.

In order to use the shorthand syntax, the scripts must have executable permissions. ls -la | ./filemod.awk

The program receives output from ls -la and disregards lines representing hidden files because dotfiles should not be renamed. The initial total line is also skipped.

If you want to invoke programs using the ./program.awk syntax, a shebang character sequence must also be specified at the top of a script pointing to your AWK binary, followed by the -f flag:

#! /usr/bin/awk -f

Alternatively, the longer invocation method is to specify the awk utility and pass a script’s filename to the -f flag explicitly:

$ ls -la | awk -f filemod.awk

Running a script this way doesn’t require it to have executable permissions, nor does the shebang character sequence have to be specified at the top of the file. However, I would recommend using the shorthand syntax because less typing is required.

Our program also accepts a variable assignment on the command line which instructs it to perform either lowercase or uppercase conversion:

$ ls -la | ./filemod.awk -v c=l # lowercase conversion

$ ls -la | ./filemod.awk -v c=u # uppercase conversion