How You can Scrape Instagram with Python
Tags: Analytics, Pandas, Python, Web Scraping, Instagram

How you can use the instaloader package in Python to scrape Instagram data

Source: Mematic (created by me)

This Article is for You

If you want to learn:

Automatically download pictures (or videos) along with their captions and other data from Instagram

How to use the instaloader package in Python

How to process your downloaded data in a clean excel file

Install Instaloader

Instaloader is a tool to download pictures (or videos) along with their captions and other metadata from Instagram.

Install Instaloader with the pip command and import the package

! pip install instaloader

import instaloader

Write your scraping function that uses the instaloader features. Replace the strings “username” & “password” with your own or a new Instagram account to avoid errors.

def get_instagram_posts(username, startdate, enddate):

# Create an instaloader object with parameters

L = instaloader.Instaloader(download_pictures = False, download_videos = False, download_comments= False, compress_json = False)



# Log in with the instaloader object

L.login("username" , "password") # Search the instagram profile

profile = instaloader.Profile.from_username(L.context, username

# Scrape the posts

posts = profile.get_posts() for post in takewhile(lambda p: p.date > startdate, dropwhile( lambda p : p.date > enddate, posts)):

print(post.date)

L.download_post(post, target = profile.username)

This function will now save all instagram posts and related data to a folder in you current working directory.

Let’s call this function on the instagram account of “realdonaldtrump”. This might take a while so be patient and let the script do its magic.

import os

import datetime # instagram username

username = "realdonaldtrump" # daterange of scraping startdate = datetime(2020, 9, 1)

enddate = datetime(2020, 10, 1) # get your current working directory current_wkdir = os.get_cwd() # Call the function. This will automatically store all the scrape data in a folder in your current working directory get_instagram_posts(username, startdate, enddate)

You notice that this data is not yet in the right format since each post has a separate json file. You will need to process all these json files to a consolidated excel file in order to perform analyses on the data.

Example of output

def parse_instafiles(username, path):

"""

This function loads in all the json files generated by the instaloader package and parses it into a csv file.

"""

#print('Entering provided directory...')

os.chdir(os.path.join(path, username))



columns = ['filename', 'datetime', 'type', 'locations_id', 'locations_name', 'mentions', 'hashtags', 'video_duration']



dataframe = pd.DataFrame(columns=[])



#print('Traversing file tree...')



glob('*UTC.json')



for file in glob('*UTC.json'):

with open(file, 'r') as filecontent:

filename = filecontent.name

#print('Found JSON file: ' + filename + '. Loading...')



try:

metadata = orjson.loads(filecontent.read())



except IOError as e:

#print("I/O Error. Couldn't load file. Trying the next one...")

continue

else:

pass

#print('Collecting relevant metadata...')

time = datetime.fromtimestamp(int(metadata['node']['taken_at_timestamp']))

type_ = metadata['node']['__typename']

likes = metadata['node']['edge_media_preview_like']['count']

comments = metadata['node']['edge_media_to_comment']['count']

username = metadata['node']['owner']['username']

followers = metadata['node']['owner']['edge_followed_by']['count']

try:

text = metadata['node']['edge_media_to_caption']['edges'][0]['node']['text']

except:

text = ""

try:

post_id = metadata['node']['id']

except:

post_id = ""

minedata = {'filename': filename, 'time': time, 'text': text,

'likes': likes, 'comments' : comments, 'username' : username, 'followers' : followers, 'post_id' : post_id}

#print('Writing to dataframe...')

dataframe = dataframe.append(minedata, ignore_index=True)

#print('Closing file...')

del metadata

filecontent.close()

#print('Storing dataframe to CSV file...')

#print('Done.')

dataframe['source'] = 'Instagram'

return dataframe

You can then use this function to process the Donald Trump Instagram data.

df_instagram = parse_instafiles(username, os.getcwd() ) df_instagram.to_excel("DonaldTrump_instagram.xlsx")

Output of excel

You now have a nice excel file with all the posts of Donald Trump in the month September until October. This format is perfect for visualisations like:

word cloud of Trump’s posts

Evolution of # likes per post

Engagement per post

…

Happy programming!

Cheers,

Ewoud