Adventures in analysing my Instagrams
Tags: Photography, Instagram, Data Visualization

Firstly a confession–I’m a fairly heavy Instagram user, and I take a lot of pictures of my dogs, food, flowers, sunsets and all the other trite things that ‘Instagrammers’ take pictures of. One of the things I hate about Instagram is that it’s really hard to search and browse your own images. You always end up needing a 3rd party app and the paging is annoying at best and at worst non existent. I wanted a way to explore at all my IGs, ideally offline on my machine so that it would act as a back up. But there’s no ‘back up’ or ‘download’ my data like say Twitter or even Flickr. Also most of the 3rd party tools to back up your IG account only grab the images and miss the titles and hashtags. Now you can go the developer API route and build a custom application to do it but as I already had all the images backed up via a great 3rd party app (Photodesk — also good for showing the details, comments etc), I decided to find a way I could use Qlik Sense to explore the images. For those who don’t know what Qlik Sense is, it’s a data analysis and visualisation tool, that can pretty much do anything with a little practice, patience and tinkering.

Between Kat and I we have approx 6500 (and counting) IGs. So first off I needed to turn 6500 images into tabular data. Which meant I simply needed to create a list of the files, and that’s as simple as Copy and Paste into a .csv file on the Mac. That gave me a way to list all the images and view them inside Qlik Sense. With a little more tinkering I could derive additional information from the IG filename. When you backup your images from Photodesk it stores them like so: 1317927813210600632_6614136.png which is the Instagram media_id. That gives you the image number and the user id. Not much but a start, as it allows me to filter by who created the image. But what I really wanted was the meta data for each IG— at least when it taken and the title. I managed to get a date for the image by extracting the creation date from the image with a simple Python script. Unfortunately this is a little misleading as no timezone info so no clear way to offset the server timestamp to the local time it was taken (still investigating that). But what this meant was that I had the beginnings of a real data table:

[Filename][image id][user id][date created]

After this first dabble with Python, I decided to use it to analyse the colour in the images. I used ColorThief for Python, I threw all 6500 images at it and created a couple more data tables mapping the dominant colour and a 4 colour palette for each image. Then I mapped those colours to the closest named colours on the CSS3 colour list, as the individual hex values are too unique for analysis. In addition I pulled out which of the major colour hues the dominant colour was in eg; red, yellow, cyan, green, blue, magenta and then black, white and grey.

[image id][Dominant colour][Dominant colour Hex][Dominant colour HLS][Closest named colour][Closest named colour Hex][major hue]

[image id][Pal 1 Hex][Pal 1 closest named colour][Pal 2 Hex][Pal 2 closest named colour][Pal 3 Hex][Pal 3 closest named colour][Pal 4 Hex][Pal 4 closest named colour]

This all associated perfectly in Qlik Sense via the [image id] and was becoming a fun and fascinating way to explore and analyse my IGs.

Next up was to get the title and that meant getting it from Instagram. To get any useful information or indeed anything out of the Instagram web service you need the shortcode for the image (eg: BJKOF0KBnS4).

https://www.instagram.com/p/BJKOF0KBnS4/

Which I didn’t have. At this point I gave up several times and almost signed up for the Instagram Developer API as it seemed impossible. But I eventually found a great post on reverse engineering the media_id (http://carrot.is/coding/instagram-ids#). This explained that the shortcode was a base64 encoding of the first part of the Instagram media_id. Simple I thought, just run it through a standard encode routine in JS or Python and save the results. Unfortunately not, after trying dozens online only 1 gave me the correct result, but I didn’t know why! Eventually I realised I needed to use a custom base62 encoding routine as IG had used a specific mapping for the alphabet and most of the built in encoders needed a string not the decimal to do the encoding. I found this routine http://stackoverflow.com/questions/1119722/base-62-conversion tweaked the map and bingo. I had the shortcodes.

This allowed me to recreate a link back to the Instagram page for each image (as above), as well as opening access to the images themselves at different sizes (t, m, l) via:

https://instagram.com/p/BJKOF0KBnS4/media/?size=t

Which means I could in theory grab the images myself or indeed reference the media from their servers, if I didn’t want to store them. But most of all it got me access to the JSON for embedding an image using the API, without needing to register. Like so:

https://api.instagram.com/oembed?url=http://instagr.am/p/BJKOF0KBnS4/

Not the nicest of things, but with a little more Python (by this time getting well and truly down and dirty with it), I could write a batch process to loop through all the shortcodes, calling the JSON and extracting just what I needed into another table (took a couple of hours, but hey, at least it didn’t reject the amount of requests I made).

[image id][Shortcode][Title][Photographer][User id]

So finally I had all my images (held locally), and a bunch of meta data about them also local stored in CSV files for easy loading into Qlik Sense. Once in Qlik Sense I could pull out additional things like hashtags (from the Title), break down the date into periods etc, and make use of the colour codes.

Unfortunately you can’t get comments, likes or location data this way, for that you need to use the authenticated API. Which after this adventure, would have been the smarter approach, but not as much fun.

Anyway here’s some of the things I found in the data.