Data Scientist’s toolkit — How to gather data from different sources | by Samarth Agrawal | Towards Data Science

Not so long ago!
Do you remember the time when data was sent to you in an external hard drive for your analysis or model building.
Now — as a data scientist, you are not limited to those means. There are several ways of storing data, sharing data as well as different sources to acquire data, augment data.
Below, I’m listing down several ways of gathering data for your analysis
CSV fileFlat File (tab, space, or any other separator)Text File (In a single file — reading data all at once)ZIP fileMultiple Text Files (Data is split over multiple text files)Download File from Internet (File hosted on a server)Webpage (scraping)APIs (JSON)Text File (Reading data line by line)RDBMS (SQL Tables)
CSV file
Flat File (tab, space, or any other separator)
Text File (In a single file — reading data all at once)
ZIP file
Multiple Text Files (Data is split over multiple text files)
Download File from Internet (File hosted on a server)
Webpage (scraping)
APIs (JSON)
Text File (Reading data line by line)
RDBMS (SQL Tables)
In Python, a file is characterized as either text or binary, and the difference between the two is important
Text files are structured as sequence of lines. Each line is terminated with a special character called EOL or End of line character. There are several types but most common are \n or ,
A Binary file type is basically any type of file that’s not a text file. Because of their nature, binary file can only be processed by an application that know or understand the file’s structure
1. CSV File
Most common format for storing and sharing dataset is comma separated format or a csv file. pandas.read_csv() is THE most useful and powerful method and I strongly recommend you to read its documentation . By using appropriate kind of sep you can load several types of data in dataframe
import pandasdf = pd.read_csv('data.csv', sep =',')
2. Flat File
but at times you might receive file that’s tab separated or a fixed width format or semi-colon separated etc. Pandas offer a variety of methods to read such data appropriately. But even read_csv works very well when you specify the right separator
import pandas# Tab separated filedf = pd.read_csv('data.tsv', sep='\t')OR # columns are separated by space df = pd.read_csv('data.txt', sep = ' ')
3. Text File
We saw that using pandas.read_csv() method we can read in txt file but let’s look at Another way of reading a txt file is Context Managers. They are very effective in allocating and managing resources, closing all the files that are open
The most widely used example of context managers is the with statement
file_name = 'data.txt'with open(file_name, mode = 'r') as file:    df = file
4. Zip File
At times, you might be given a csv file which in turn may contain the csv file you need in order to conserve size. To extract .zip file from python you need to use ‘zipfile’ library. If you are using windows doing this will be a far better approach than extracting it from windows
zipfile is also a Context Manager and will support the with statement as we have seen above
from zipfile import ZipFilefile_name = 'my_zip_file.zip'with open(file_name, mode = 'r') as f:    f.extractall()# Extractall method will extract all the contents of zipfile in the same folder# now we can load the extracted csv file in our dataframeimport pandasdf = pd.read_csv('my_csv_file.csv')
5. Multiple Text Files
You might come across a situation where your data is provided to you in the form of multiple text files. For ex, you are provided reviews of 1000 movie titles and instead of putting all into one they have put each individual review as one file and 1000 such text files are provided
We will use a library glob — which makes opening files with similar path structures simple
import pandasimport globfolder_name = 'my_folder'df_list = []for review in glob.glob('my_folder/*.txt'):    with open(review, mode = 'r') as file:        movie = {}        movie['title'] = file.readline()        movie['review'] = file.read()df_list.append(movie)df = pd.DataFrame(df_list)
is a wildcard in the glob statement. It allows python to scan for all the .txt files in the given path
6. Download files from Internet
If you have to download a file kept on a server. You will have to use a library — requests
import requestsimport ossample_url = '192.145.232.xx/2019/2/class_notes/gather_data.txt'folder_name = 'my_folder'if not os.path.exists(folder_name):    os.makedirs(folder_name)response = requests.get(url)file_name = 'gather_data.txt'file_loc = os.path.join(folder_name, file_name)with open(file_loc, mode='wb') as outfile:    outfile.write(response.content)
7. Web page (Web Scraping)
Web scraping is a fancy way of extracting data from the web pages using code. Data that lives on the web pages is called HTML, Hyper Text Markup Language. It’s made up of these things called tags which gives the web page structure. <head>, <title>, <div>, <h1> ..... etc
Because HTML code is just text, these tags and content within them can be extracted using a parser. For this we will use a library — BeautifulSoup
Let’s say we want to extract information about Avengers Endgame (2019) from IMDb and rotten tomatoes both
Note: Inspect element is your best friend here in finding the relevant tags and attributes for extracting the data
from bs4 import BeautifulSoup as bsimport requestsimport pandasimdb_url = 'https://www.imdb.com/title/tt4154796/'response = requests.get(imdb_url)soup = bs(response.content, features = b'lxml)movie = {}movie['title'] = soup.find('div', attrs = {'class': 'title_wrapper'}).find('h1').textmovie['imdb_rating'] = soup.find('span', attrs = {'itemprop': 'ratingValue'}).textrt_url = 'https://www.rottentomatoes.com/m/avengers_endgame'response = requests.get(rt_url)soup = bs(response.content, features = 'lxml')movie['tomatometer'] = soup.find('span', attrs = {'class' : 'mop-ratings-wrap__percentage'}).textmovie['audience_score'] = soup.find('div', attrs = {'class' : 'audience-score'}).find('span', attrs={'class' : 'mop-ratings-wrap__percentage'}).textdf = pd.DataFrame(movie)
This way we have collected information on Avengers Endgame from 2 different sources
8. APIs (Application Programming Interfaces)
You can ofcourse extract information from webpage but a better way of accessing information is through an API.
Wikipedia has several publicly open API, one of the popular one is Mediawiki. We will use python library wptools
import wptoolsimport pandas# For a wikipedia URL 'https://en.wikipedia.org/wiki/Avengers:_Endgame' we only need to pass the string after /wiki/wiki_page = wptools.page('Avengers:_Endgame).get()# Now this wiki_page has fetched extracts, images, infobox data, wiki data etc# By using wikipage.data() method we can extract all the informationwiki_page.data['image'] # this will return 3 images backfirst_image = wiki_page.data['image'][0]print (first_image['url'])'https://upload.wikimedia.org/wikipedia/en/0/0d/Avengers_Endgame_poster.jpg'# Now you can save this poster link or use the requests method as seen above to save poster of the movie
9. Text Files (Reading data line by line)
Let’s say you have downloaded tweets data using twitter api — tweepy
And you have saved all the tweets in a text file — tweets_json.txt
Your objective now is to extract useful information from these tweets, save it in a dataframe and use it for further analysis. This is a combination of skills you learned in step 3 and 8
Part 1 — Let me show you how to get twitter data using tweepy.
import tweepyimport jsonconsumer_key = 'xxx'consumer_secret = 'xxx'access_token = 'xxx'access_secret = 'xxx'auth = OAuthHandler(consumer_key, consumer_secret)auth.set_access_token(access_token, access_secret)api = tweepy.API(auth, wait_on_rate_limit = True)tweet_ids = [] #This will be the list of tweets for which you need the information forfile_name = 'tweets_json.txt'with open(file_name, mode='w') as file:    for tweet_id in tweet_id:        tweet = api.get_status(tweet_id, tweet_mode = 'extended')        json.dump(tweet._json, file)        file.write('/n')
Part 2 — Reading this tweets_json.txt to extract useful information
import pandasimport jsondf_list = []with open('tweets_json.txt', mode = 'r') as file:    for line in file:        tweet = {}        json_data = json.loads(line)        tweet['tweet_id'] = json_data['id']        tweet['retweet_count'] = json_data['retweet_count']        tweet['favorite_count'] = json_data['favorite_count']        df_list.append(tweet)tweet_df = pd.DataFrame(df_list)
10. RDBMS (SQL Database)
A database is “ a structured set of data held in a computer, especially one that is accessible in various ways.” It facilitates storage, retrieval, modification and deletion of data
In the context of data wrangling, databases and SQL come into play for storing data or gathering data:
Connecting to database and importing data into a pandas Dataframe
Connecting to database and storing data from a pandas Dataframe to database
Connecting to a database in python —
Connecting to a database in python —
We will connect to SQLite database using SQLAlchemy, a database toolkit for python
import sqlalchemyimport pandasengine = sqlalchemy.create_engine('sqlite:///movies.db')
movies.db will not show up in the jupyter notebook dashboard yet
2. Store Pandas Dataframe in a database
# Store dataframe 'df' in a table called movie_tbl in the databasedf.to_sql('movie_tbl', engine, index=False)
movies.db will now show in the jupyter notebook dashboard
3. Read database table in a pandas dataframe
df = pd.read_sql('SELECT * FROM movie_tbl', engine)
I have tried to give a basic idea of wide variety of things and avoided writing down so many details
Please feel free to write your thoughts / suggestions / feedback