Everything You Need to Start Text Processing in UNIX and Linux!
Channel: The Expat Professor
Date: 2023-02-08

This tutorial covers everything that you must know to get started performing data analysis or text processing in Linux or UNIX. UNIX was created as a simple system that allowed users to build pipelines of commands to achieve enormous tasks. In this lesson, we cover the most commonly used data analysis tools.

The Commands used in this video:
---------------------------------
File Operations:
mv

System Operations:
alias

Data Analysis:
vi
cat
grep
awk
sed
sort
uniq

Special Conventions:
Backslash Escape = \
Beginning of a line = ^
End of a line = $
Range of Numbers = [0-9] or [0-3]
Tab = \t
Column Reference in Awk = $1 $2 $3, et cetera...



The Command History used in this video:
---------------------------------------
vim today.txt
cat today.txt 
mv today.txt today.csv
alias grep
cat today.csv | grep New
cat today.csv
cat today.csv | grep -v New
cat today.csv | grep -v ew
cat today.csv | grep ew
cat today.csv | grep w
cat today.csv | grep y
cat today.csv | grep Y
cat today.csv | grep -i y
cat today.csv | grep "^ $"
cat today.csv | grep -v "^ $"
cat today.csv | grep -v "^$"
cat today.csv | grep "^$"
vi today.csv 
cat today.csv | grep -i "s"
cat today.csv | grep -i "^s"
cat today.csv | grep -i "Jose"
cat today.csv | grep -i "se"
cat today.csv | grep -i "5"
cat today.csv | grep -i "3"
cat today.csv | awk '{ print $1 }'
cat today.csv | awk -F"," '{ print $1 }'
cat today.csv | awk -F"," '{ print $2 }'
cat today.csv | grep "[3-2]*"
cat today.csv | grep "[3..2]*"
cat today.csv | grep "[0..3]*"
cat today.csv | grep "[0-3]*"
cat today.csv | grep ",[0-3]"
cat today.csv | grep ",[0-3]" | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }'
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }'
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort 
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n | sed s/ //g
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n | sed s/" "//g
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n | sed s/\ //g
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n | sed s/" "/_/g
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n | sed s/" "/_/g | sed s/:_/" "/g 
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n | sed s/" "/_/g | sed s/:_/:" "/g 
cat today.csv | awk -F"," '{ print "Temp: " $2 "\tCity: " $1 }' | sort -n | sed s/" "/_/g | sed s/:_/:" "/g | uniq
vim file.list
cat file.list | sort
cat file.list | sort | grep -v "^$"
cat file.list | grep -v "^$" 
cat file.list | grep -v "^$" | sort
cat file.list | grep -v "^$" | uniq
cat file.list | grep -v "^$" | sort 
cat file.list | grep -v "^$" | sort | uniq
cat file.list | grep -v "^$" | sort | uniq -c
cat file.list | grep -v "^$" | sort | uniq -c | sort -n
cat file.list | sort | uniq -c | sort -n




The Files used in the video:
----------------------------
File 01: today.csv 
San Francisco,43
San Jose,56
New York,23
Pittsburgh,30
Abu Dhabi,78

File 02: file.list
3
3
7
7
7
3
3
4
5
7
6
4
6
66

7


9
2
9
9
9



-----------------------
Thank you for dropping by!

Please join me:
 https://www.youtube.com/PaulWPoteete
 https://www.twitter.com/PaulWPoteete
 https://www.linkedin.com/in/PaulWPoteete
CC by Paul W. Poteete