csirtgadgets/iodef-pb-simple-perl — pull request #8 by mjbroekman (2013-11-07 14:52:50+00:00)
Title: Modified csv plugin to allow for skipping blank rows and de-duplication of results

Added two options "suppress_blanks" and "unique_only" to the csv plugin to allow users to suppress blank rows and get unique entries instead of all entries.

This came from running queries such as:
$ cif -c 0 -q example.com -p csv -f assessment,description
# assessment,description
search,search example.com
search,search example.com
search,search example.com
search,search example.com
search,search example.com
search,search example.com
search,search example.com

If this is the data I'm looking for ("search,search example.com"), I don't need all 7 entries (one each for the various detect/report times).

Running the previous command with the 'unique_only' command (added as -u in the cif command), the output looks like this:
$ cif -C /home/cif/.cif -c 0 -q example.com -p csv -f assessment,description -u
# assessment,description
search,search example.com

Similarly, if I'm looking for Google IP addresses, I can use the following query:
$ cif -c 0 -q google.com -p csv -f rdata

This gives 51 lines of output (header + 50 rows of data). However, there are only 9 IP addresses in the rdata field. The other results have no data in the rdata field and, as a result, you get blank lines.

Running the same query with the 'suppress_blanks' option (added in the cif command as -B), you get the following output:
$ cif -c 0 -q google.com -p csv -f rdata -B
# rdata
216.239.34.10
216.239.38.10
216.239.36.10
216.239.32.10
216.239.36.10
216.239.34.10
216.239.38.10
205.158.11.53

The code retained the closest performance to the original plugin in terms of execution time.