Unix & Linux: Count distinct values of a field in a file (3 Solutions!!)
Channel: Roel Van de Paar
Date: 16/08/2020

Unix & Linux: Count distinct values of a field in a file


The Question: I have a file contains around million number of lines. In the lines I have a
field called transactionid, which has repetitive values. What I need to do is
to count them distinctly.
No matter of how many times a value is repeated, it should be counted only
once.

Solutions: Please watch the whole video to see all solutions, in order of how many people found them helpful

== This solution helped 3 people ==
Maybe not the sleekest method, but this should work:
awk '{print $1}' your_file | sort | uniq | wc -l
where $1 is the number corresponding to the field to be parsed.

== This solution helped 27 people ==
OK, Assuming that your file is a text file, having the fields separated by
comma separator ','. You would also know which field 'transactionid' is in
terms of its position. Assuming that your 'transactionid' field is 7th field.
awk -F ',' '{print $7}' text_file | sort | uniq -c
This would count the distinct/unique occurrences in the 7th field and prints
the result.

== This solution helped 3 people ==
There is no need to sort the file .. (uniq requires the file to be sorted)
This awk script assumes the field is the first whitespace delimiited field.
With thanks & praise to God, and with thanks to the many people who have made this project possible! | Content (except music & images) licensed under cc by-sa 3.0 | Music: https://www.bensound.com/royalty-free-music | Images: https://stocksnap.io/license & others | With thanks to user Rui F Ribeiro (https://unix.stackexchange.com/users/138261), user phk (https://unix.stackexchange.com/users/117599), user Peter.O (https://unix.stackexchange.com/users/2343), user Olgun Kaya (https://unix.stackexchange.com/users/6115), user Nikhil Mulley (https://unix.stackexchange.com/users/12856), and the Stack Exchange Network (http://unix.stackexchange.com/questions/28845). Trademarks are property of their respective owners. Disclaimer: All information is provided "AS IS" without warranty of any kind. You are responsible for your own actions. Please contact me if anything is amiss at Roel D.OT VandePaar A.T gmail.com.