gawk - Unix, Linux Command

gawk - pattern scanning and processing language.
gawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
gawk [ POSIX or GNU style options ] [ -- ] program-text file ...

pgawk [ POSIX or GNU style options ] -f program-file [ -- ] file ...
pgawk [ POSIX or GNU style options ] [ -- ] program-text file ...

Pgawk is the profiling version of
gawk. It is identical in every way to
gawk, except that programs run more slowly,
and it automatically produces an execution profile in the file
awkprof.out when done.
See the
--profile option, below.


The command line consists of options to
gawk itself, the AWK program text (if not supplied via the
-f or
--file options), and values to be made
available in the
ARGC and
ARGV pre-defined AWK variables.


Following the POSIX standard,
gawk-specific options are supplied via arguments to the
-W option.  Multiple
-W options may be supplied
Each
-W option has a corresponding long option, as detailed below.
Arguments to long options are either joined with the option
by an
= sign, with no intervening spaces, or they may be provided in the
next command line argument.
Long options may be abbreviated, as long as the abbreviation
remains unique.


Having a list of all the global variables is a good way to look for
typographical errors in your programs.
You would also use this option if you have a large program with a lot of
functions, and you want to be sure that your functions don’t
inadvertently use global variables that you meant to be local.
(This is a particularly easy mistake to make with simple variable
names like
i, j, and so on.)


pattern { action statements }

function name(parameter list) { statements }


Gawk first reads the program source from the
program-file(s) if specified,
from arguments to
--source, or from the first non-option argument on the command line.
The
-f and
--source options may be used multiple times on the command line.
Gawk reads the program text as if all the
program-files and command line source texts
had been concatenated together.  This is useful for building libraries
of AWK functions, without having to include them in each new AWK
program that uses them.  It also provides the ability to mix library
functions with command line programs.


The environment variable
AWKPATH specifies a search path to use when finding source files named with
the
-f option.  If this variable does not exist, the default path is
".:/usr/local/share/awk".
(The actual directory may vary, depending upon how
gawk was built and installed.)
If a file name given to the
-f option contains a ‘‘/’’ character, no path search is performed.


Gawk executes AWK programs in the following order.
First,
all variable assignments specified via the
-v option are performed.
Next,
gawk compiles the program into an internal form.
Then,
gawk executes the code in the
BEGIN block(s) (if any),
and then proceeds to read
each file named in the
ARGV array.
If there are no files named on the command line,
gawk reads the standard input.


If a filename on the command line has the form
var=val it is treated as a variable assignment.  The variable
var will be assigned the value
val. (This happens after any
BEGIN block(s) have been run.)
Command line variable assignment
is most useful for dynamically assigning values to the variables
AWK uses to control how input is broken into fields and records.
It is also useful for controlling state if multiple passes are needed over
a single data file.


If the value of a particular element of
ARGV is empty (""),
gawk skips over it.


For each record in the input,
gawk tests to see if it matches any
pattern in the AWK program.
For each pattern that the record matches, the associated
action is executed.
The patterns are tested in the order they occur in the program.


Finally, after all the input is exhausted,
gawk executes the code in the
END block(s) (if any).


As each input record is read,
gawk splits the record into
fields, using the value of the
FS variable as the field separator.
If
FS is a single character, fields are separated by that character.
If
FS is the null string, then each individual character becomes a
separate field.
Otherwise,
FS is expected to be a full regular expression.
In the special case that
FS is a single space, fields are separated
by runs of spaces and/or tabs and/or newlines.
(But see the discussion of
--posix, below).
NOTE: The value of
IGNORECASE (see below) also affects how fields are split when
FS is a regular expression, and how records are separated when
RS is a regular expression.


If the
FIELDWIDTHS variable is set to a space separated list of numbers, each field is
expected to have fixed width, and
gawk splits up the record using the specified widths.  The value of
FS is ignored.
Assigning a new value to
FS overrides the use of
FIELDWIDTHS, and restores the default behavior.


Each field in the input record may be referenced by its position,
$1, $2, and so on.
$0 is the whole record.
Fields need not be referenced by constants:


n = 5

print $n


prints the fifth field in the input record.


The variable
NF is set to the total number of fields in the input record.


References to non-existent fields (i.e. fields after
$NF) produce the null-string.  However, assigning to a non-existent field
(e.g.,
$(NF+2) = 5) increases the value of
NF, creates any intervening fields with the null string as their value, and
causes the value of
$0 to be recomputed, with the fields being separated by the value of
OFS. References to negative numbered fields cause a fatal error.
Decrementing
NF causes the values of fields past the new value to be lost, and the value of
$0 to be recomputed, with the fields being separated by the value of
OFS. 

Assigning a value to an existing field
causes the whole record to be rebuilt when
$0 is referenced.
Similarly, assigning a value to
$0 causes the record to be resplit, creating new
values for the fields.


Gawk’s built-in variables are:




Thus, if
IGNORECASE is not equal to zero,
/aB/ matches all of the strings "ab", "aB", "Ab",
and "AB".
As with all AWK variables, the initial value of
IGNORECASE is zero, so all regular expression and string
operations are normally case-sensitive.
Under Unix, the full ISO 8859-1 Latin-1 character set is used
when ignoring case.
As of
gawk 3.1.4, the case equivalencies are fully locale-aware, based on
the C
<ctype.h> facilities such as
isalpha(), and
tolupper(). 

Arrays are subscripted with an expression between square brackets
([ and ]). If the expression is an expression list
(expr, expr ...) then the array subscript is a string consisting of the
concatenation of the (string) value of each expression,
separated by the value of the
SUBSEP variable.
This facility is used to simulate multiply dimensioned
arrays.  For example:


i = "A"; j = "B"; k = "C"

x[i, j, k] = "hello, world\n"


assigns the string "hello, world\n" to the element of the array
x which is indexed by the string "A\034B\034C".  All arrays in AWK
are associative, i.e. indexed by string values.


The special operator
in may be used in an
if or
while statement to see if an array has an index consisting of a particular
value.



if (val in array)
        print array[val]


If the array has multiple subscripts, use
(i, j) in array. 

The
in construct may also be used in a
for loop to iterate over all the elements of an array.


An element may be deleted from an array using the
delete statement.
The
delete statement may also be used to delete the entire contents of an array,
just by specifying the array name without a subscript.


Variables and fields
may be (floating point) numbers, or strings, or both.  How the
value of a variable is interpreted depends upon its context.  If used in
a numeric expression, it will be treated as a number, if used as a string
it will be treated as a string.


To force a variable to be treated as a number, add 0 to it; to force it
to be treated as a string, concatenate it with the null string.


When a string must be converted to a number, the conversion is accomplished
using
strtod(3).
A number is converted to a string by using the value of
CONVFMT as a format string for
sprintf(3),
with the numeric value of the variable as the argument.
However, even though all numbers in AWK are floating-point,
integral values are
always converted as integers.  Thus, given



CONVFMT = "%2.2f"
a = 12
b = a ""


the variable
b has a string value of "12" and not "12.00".


Gawk performs comparisons as follows:
If two variables are numeric, they are compared numerically.
If one value is numeric and the other has a string value that is a
‘‘numeric string,’’ then comparisons are also done numerically.
Otherwise, the numeric value is converted to a string and a string
comparison is performed.
Two strings are compared, of course, as strings.
Note that the POSIX standard applies the concept of
‘‘numeric string’’ everywhere, even to string constants.
However, this is
clearly incorrect, and
gawk does not do this.
(Fortunately, this is fixed in the next version of the standard.)


Note that string constants, such as "57", are
not numeric strings, they are string constants.
The idea of ‘‘numeric string’’
only applies to fields,
getline input,
FILENAME, ARGV elements,
ENVIRON elements and the elements of an array created by
split() that are numeric strings.
The basic idea is that
user input, and only user input, that looks numeric,
should be treated that way.


Uninitialized variables have the numeric value 0 and the string value ""
(the null, or empty, string).


String constants in AWK are sequences of characters enclosed
between double quotes (").  Within strings, certain
escape sequences are recognized, as in C.  These are:




In compatibility mode, the characters represented by octal and
hexadecimal escape sequences are treated literally when used in
regular expression constants.  Thus,
/a\52b/ is equivalent to /a\*b/.


{ print } 

which prints the entire record.


Comments begin with the ‘‘#’’ character, and continue until the
end of the line.
Blank lines may be used to separate statements.
Normally, a statement ends with a newline, however, this is not the
case for lines ending in
a ‘‘,’’,
{, ?, :, &&, or
||. Lines ending in
do or
else also have their statements automatically continued on the following line.
In other cases, a line can be continued by ending it with a ‘‘\’’,
in which case the newline will be ignored.


Multiple statements may
be put on one line by separating them with a ‘‘;’’.
This applies to both the statements within the action part of a
pattern-action pair (the usual case),
and to the pattern-action statements themselves.



BEGIN 
END 
/regular expression/ 
relational expression 
pattern && pattern 
pattern || pattern 
pattern ? pattern : pattern 
(pattern) 
! pattern 
pattern1, pattern2 


BEGIN and
END are two special kinds of patterns which are not tested against
the input.
The action parts of all
BEGIN patterns are merged as if all the statements had
been written in a single
BEGIN block.  They are executed before any
of the input is read.  Similarly, all the
END blocks are merged,
and executed when all the input is exhausted (or when an
exit statement is executed).
BEGIN and
END patterns cannot be combined with other patterns in pattern expressions.
BEGIN and
END patterns cannot have missing action parts.


For
/regular expression/ patterns, the associated statement is executed for each input record that matches
the regular expression.
Regular expressions are the same as those in
egrep(1),
and are summarized below.


A
relational expression may use any of the operators defined below in the section on actions.
These generally test whether certain fields match certain regular expressions.


The
&&, ||, and
! operators are logical AND, logical OR, and logical NOT, respectively, as in C.
They do short-circuit evaluation, also as in C, and are used for combining
more primitive pattern expressions.  As in most languages, parentheses
may be used to change the order of evaluation.


The
?: operator is like the same operator in C.  If the first pattern is true
then the pattern used for testing is the second pattern, otherwise it is
the third.  Only one of the second and third patterns is evaluated.


The
pattern1, pattern2 form of an expression is called a
range pattern. It matches all input records starting with a record that matches
pattern1, and continuing until a record that matches
pattern2, inclusive.  It does not combine with any other sort of pattern expression.


Interval expressions are only available if either
--posix or
--re-interval is specified on the command line.


Character classes are a new feature introduced in the POSIX standard.
A character class is a special notation for describing
lists of characters that have a specific attribute, but where the
actual characters themselves can vary from country to country and/or
from character set to character set.  For example, the notion of what
is an alphabetic character differs in the USA and in France.


A character class is only valid in a regular expression
inside the brackets of a character list.  Character classes consist of
[:, a keyword denoting the class, and
:]. The character
classes defined by the POSIX standard are:


Two additional special sequences can appear in character lists.
These apply to non-ASCII character sets, which can have single symbols
(called
collating elements) that are represented with more than one
character, as well as several characters that are equivalent for
collating, or sorting, purposes.  (E.g., in French, a plain ‘‘e’’
and a grave-accented e` are equivalent.)


The operators in AWK, in order of decreasing precedence, are




The control statements are
as follows:



if (condition) statement [ else statement ]
while (condition) statement 
do statement while (condition)
for (expr1; expr2; expr3) statement
for (var in array) statement
break
continue
delete array[index]
delete array
exit [ expression ]
{ statements }


The input/output statements are as follows:




NOTE: If using a pipe or co-process to
getline, or from
print or
printf within a loop, you
must use
close() to create new instances of the command.
AWK does not automatically close pipes or co-processes when
they return EOF.


The AWK versions of the
printf statement and
sprintf() function
(see below)
accept the following conversion specification formats:


Optional, additional parameters may lie between the
% and the control letter:


When doing I/O redirection from either
print or
printf into a file,
or via
getline from a file,
gawk recognizes certain special filenames internally.  These filenames
allow access to open file descriptors inherited from
gawk’s parent process (usually the shell).
These file names may also be used on the command line to name data files.
The filenames are:


print "You blew it!" > "/dev/stderr"


whereas you would otherwise have to use


print "You blew it!" | "cat 1>&2"


The following special filenames may be used with the
|& co-process operator for creating TCP/IP network connections.


AWK has the following built-in arithmetic functions:




Gawk has the following built-in string functions:






The default
domain is the value of
TEXTDOMAIN. If
directory is the null string (""), then
bindtextdomain() returns the current binding for the
given
domain. 

If you supply a value for
category, it must be a string equal to
one of the known locale categories described
in GAWK: Effective AWK Programming.
You must also supply a text domain.  Use
TEXTDOMAIN if you want to use the current domain.


If you supply a value for
category, it must be a string equal to
one of the known locale categories described
in GAWK: Effective AWK Programming.
You must also supply a text domain.  Use
TEXTDOMAIN if you want to use the current domain.


function name(parameter list) { statements }


Functions are executed when they are called from within expressions
in either patterns or actions.  Actual parameters supplied in the function
call are used to instantiate the formal parameters declared in the function.
Arrays are passed by reference, other variables are passed by value.


Since functions were not originally part of the AWK language, the provision
for local variables is rather clumsy: They are declared as extra parameters
in the parameter list.  The convention is to separate local variables from
real parameters by extra spaces in the parameter list.  For example:



function  f(p, q,     a, b)     # a and b are local
{
        ...
}

/abc/   { ... ; f(1, 2) ; ... }


/abc/   { ... ; f(1, 2) ; ... }


The left parenthesis in a function call is required
to immediately follow the function name,
without any intervening white space.
This is to avoid a syntactic ambiguity with the concatenation operator.
This restriction does not apply to the built-in functions listed above.


Functions may call each other and may be recursive.
Function parameters used as local variables are initialized
to the null string and the number zero upon function invocation.


Use
return expr to return a value from a function.  The return value is undefined if no
value is provided, or if the function returns by ‘‘falling off’’ the
end.


If
--lint has been provided,
gawk warns about calls to undefined functions at parse time,
instead of at run time.
Calling an undefined function at run time is a fatal error.


The word
func may be used in place of function.



Print and sort the login names of all users:

        BEGIN   { FS = ":" }
                { print $1 | "sort" }

Count lines in a file:

                { nlines++ }
        END     { print nlines }

Precede each line by its number in the file:

        { print FNR, $0 }

Concatenate and line number (a variation on a theme):

        { print NR, $0 }
Run an external command for particular lines of data:

        tail -f access_log |
        awk ’/myhome.html/ { system("nmap " $1 ">> logdir/myhome.html") }’


        BEGIN   { FS = ":" }
                { print $1 | "sort" }


Count lines in a file:


                { nlines++ }
        END     { print nlines }


Precede each line by its number in the file:


        { print FNR, $0 }


Concatenate and line number (a variation on a theme):


        { print NR, $0 }
Run an external command for particular lines of data:


        tail -f access_log |
        awk ’/myhome.html/ { system("nmap " $1 ">> logdir/myhome.html") }’


String constants are sequences of characters enclosed in double
quotes.  In non-English speaking environments, it is possible to mark
strings in the AWK program as requiring translation to the native
natural language. Such strings are marked in the AWK program with
a leading underscore (‘‘_’’).  For example,


gawk ’BEGIN { print "hello, world" }’


always prints
hello, world. But,


gawk ’BEGIN { print _"hello, world" }’


might print
bonjour, monde in France.


There are several steps involved in producing and running a localizable
AWK program.


   BEGIN { TEXTDOMAIN = "myprog" }


This allows
gawk to find the
.mo file associated with your program.
Without this step,
gawk uses the
messages text domain,
which likely does not contain translations for your program.


The book indicates that command line variable assignment happens when
awk would otherwise open the argument as a file, which is after the
BEGIN block is executed.  However, in earlier implementations, when such an
assignment appeared before any file names, the assignment would happen
before the
BEGIN block was run.  Applications came to depend on this ‘‘feature.’’
When
awk was changed to match its documentation, the
-v option for assigning variables before program execution was added to
accommodate applications that depended upon the old behavior.
(This feature was agreed upon by both the Bell Laboratories and the GNU developers.)


The
-W option for implementation specific features is from the POSIX standard.


When processing arguments,
gawk uses the special option ‘‘--’’ to signal the end of
arguments.
In compatibility mode, it warns about but otherwise ignores
undefined options.
In normal operation, such arguments are passed on to the AWK program for
it to process.


The AWK book does not define the return value of
srand(). The POSIX standard
has it return the seed it was using, to allow keeping track
of random number sequences.  Therefore
srand() in
gawk also returns its current seed.


Other new features are:
The use of multiple
-f options (from MKS
awk); the
ENVIRON array; the
\a, and
\v escape sequences (done originally in
gawk and fed back into the Bell Laboratories version); the
tolower() and
toupper() built-in functions (from the Bell Laboratories version); and the ANSI C conversion specifications in
printf (done first in the Bell Laboratories version).


a = length      # Holy Algol 60, Batman!


is the same as either of


a = length()

a = length($0)


This feature is marked as ‘‘deprecated’’ in the POSIX standard, and
gawk issues a warning about its use if
--lint is specified on the command line.


The other feature is the use of either the
continue or the
break statements outside the body of a
while, for, or
do loop.  Traditional AWK implementations have treated such usage as
equivalent to the
next statement.
Gawk supports this usage if
--traditional has been specified.


The following features of
gawk are not available in
POSIX
awk. 

When
gawk is invoked with the
--traditional option,
if the
fs argument to the
-F option is ‘‘t’’, then
FS is set to the tab character.
Note that typing
gawk -F\t ... simply causes the shell to quote the ‘‘t,’’, and does not pass
‘‘\t’’ to the
-F option.
Since this is a rather ugly special case, it is not the default behavior.
This behavior also does not occur if
--posix has been specified.
To really get a tab character as the field separator, it is best to use
single quotes:
gawk -F’\t’ .... 

If
gawk is
configured with the
--enable-switch option to the
configure command, then it accepts an additional control-flow statement:

switch (expression) {
case value|regex : statement
...
[ default: statement ]
}


If
POSIXLY_CORRECT exists in the environment, then
gawk behaves exactly as if
--posix had been specified on the command line.
If
--lint has been specified,
gawk issues a warning message to this effect.

Basic function of gawk is to search for pattern(s) in each line of input and perform specific task when the pattern is found. 
The input to gawk can be either file, standard input (keyboard) or even output of another command/program. 
A gawk program is a series of rules with each rule specifying a pattern to search for, followed by the action to take when the pattern is found.
Below is the basic syntax of gawk rule:

pattern { action }

1. One-shot throw-away programs
This is useful when your program may have few instructions and you don't want to create a separate for this program.
Below command prints the lines from /etc/password file where the string "root" appears.

$ gawk '/root/ { print }' /etc/passwd

root:x:0:0:root:/root:/bin/bash
operator:x:11:0:operator:/root:/sbin/nologin

$ gawk '{split($0,a,":"); print a[1]}' /etc/passwd

root
bin
daemon
adm
...
...

2. Without input files
One can run gawk without any input files. In such case input for program will be taken from standard input.
Below example is same as above but only difference here is that input comes from STDIN (or keyboard)

$ gawk '/root/ { print }'

root test      => This is user input
root test      => This is ouput from program as there is match for "root"
simple test    => This is user input. No match for "root"
complex test   => This is user input. No match for "root"

3. Running long programs
When the program is big and runs into several instructions, it is convenient to put it into separate file.
By convention the the gawk program file can be named as ".awk" but it really does not matter whether it has any extension or not.

If the program is in separate file, it has to be executed as shown below:

gawk -f program-file input-file1 input-file-2...

{
        split($0, a, ":")
        if ( a[1] == "root" )
                print
}

$ gawk -f test.awk /etc/passwd

root:x:0:0:root:/root:/bin/bash

4. Self contained gawk scripts
The script will have the gawk interpreter specified within itself and can be directly invoked without using explicit gawk on the command line.
The script starts with "#!" followed by the full path of the gawk binary on the system.
Consider below program file: test.awk

#!/bin/gawk -f

{
        split($0, a, ":")
        if ( a[1] == "root" )
                print
}

$ chmod +x test.awk
$ ls -l test.awk
-rwxr-xr-x 1 root root 70 Jan 26 13:06 test.awk

$ ./test.awk /etc/passwd

root:x:0:0:root:/root:/bin/bash

#!/bin/gawk -f

BEGIN {
        print "START: Initialization activity called here"
}
{
        print
}
END {
        print "END: Cleanup called here"
}

$ ./test.awk /etc/passwd

START: Initialization activity called here
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
...
...
END: Cleanup called here

#!/bin/gawk -f

function printName(name)
{
        print "My name is "name
}

BEGIN {
        print "START"

        printName("John");
        printName("Tony");
}
END {
        print "END"
}

$ ./test.awk /etc/passwd
START
My name is John
My name is Tony
END

#!/bin/gawk -f

BEGIN {
        print "START"
        $totalLines = 0;
}
{
        totlaLines++;
        # Do some processing here
}
END {
        print "Total Lines processed: "totlaLines
        print "END"
}

$ ./test.awk /etc/passwd

START
Total Lines processed: 35