Merge pull request #9 from annajordanous/master

Code rewritten for large-scale data collection.
Merge pull request #10 from annajordanous/master

Code rewritten for large-scale data collection.
minimal data collector in progress

get_minimal will collect a snowball sample with less data about each
user, in order to crawl through the network faster
get_minimal works as data collector

doesn't write data yet
bug fix for get_minimal

as above
Minimal snowball sampler now works

This now collects data and saves it to an SQLite database. It uses a
different approach than get_soundcloud_data: first, it only requests
followers and followings, and bases the users and x_follows_y on those,
ignoring anything else; second, it works in 'steps', basically the
number of degrees of separation from a starting individual. The idea is
that because we only get up to 50 follows and followings, we may need
to keep re-sampling from the same starting points, but without
necessarily wanting to go many degrees of separation away from them
until we've covered their immediate networks (say, when the number of
followers for each individual is higher than a certain proportion of
the total that exist). So the next thing will be to write the code that
queries the database and decides who to re-request data on. Also to
start calling some of the other helpful functions in
get_soundcloud_data.py, for example to introduce a time delay. At the
moment, I've tried running this with just two steps from one of our
interviewees, just once - I got 4264 follow relationships between 3185
individuals with 101 calls to the Soundcloud API, which took well under
a minute (the max would have been 10001 individuals, in the same amount
of time).
get_minimal now works from a database

As I have found, the Soundcloud API returns a maximum of 50 results for
most (maybe all) calls. get_minimal.py now reads from the sqlite
database to judge whether to re-collect accounts (based on whether a
certain threshold level has been reached in terms of how much data has
been collected from them and how much remains to be collected).
Unfortunately, this seems to be pointless as tests suggest that you get
the same 50 results every time you execute the same API call (or at
least, that is what was happening last night). To do: add calls to
functions in get_soundcloud_data.py to create a log, add time delay,
etc.

Other change: genre_relationships.py now has a slightly more elegant
way of checking for whether tables are present. This more generalised
function is now also called by get_minimal.py.
comments_corp updated for new column names

'user' and 'track' updated to 'user_id' and 'track_id' in favesp()
function within comments_corp.py
get_minimal_lomem.py added

get_minimal_lomem is like get_minimal.py except that it uses text files
instead of RAM to hold the lists of user IDs that it works through (it
also has logging and time delay based on get_soundcloud_data.py,
although there's a difference with the time delay in that once it
reaches max_attempts it starts taking very long - and successively
longer - delays until interrupted).
Merge branch 'master' of https://github.com/ValuingElectronicMusic/network-analysis
bug fixes for get_minimal_lomem

removed the results of some careless cut-and-pasting
ongoing work with detect_english

now using most common 5k words in british national corpus instead of
OpenOffice spelling dictionaries (faster - and less liable to be
confused by e.g. Italian loanwords)
Added .txt files to gitignore

Can undo this later. Should change the module that's producing them to
produce e.g. .tmp instead.
forward_lomem.py added

New network crawler: forward_lomem.py. This one moves forward through
the network from an individual to those he/she follows, and then
forward, making sure it has got ALL the followings of everyone at each
stage before moving on to the next stage.
Ignore .tmp files, don't ignore .txt

The forward_lomem.py crawler now generates .tmp files rather than .txt
files, so the latter can now be removed from .gitignore
foward_lomem.py redundant code removed

Removed all code pertaining to collection of followers (left over from
get_minimal_lomem.py). It now only collects followings - see comments
at head of file for explanation of why this is done.
forward_waves.py now seems to work!

This is a new program for crawling SoundCloud. It only crawls forward
from follower to followed, and it does so in waves, each corresponding
to a specific degree of separation from a seed individual. It collects
all individuals in a single wave before moving to the next. Holds as
little data as possible in memory and as much as possible in .tmp files
and the sqlite database to which it's adding. Data collected is held in
buffers of a certain size; when they're filled, they're cleared and
contents is sent to the sqlite database. If interrupted, has a resume
function for picking up where it left off.
Working on genres & tags again

As above.
add .zip to .gitignore

as above
Merge branch 'master' of https://github.com/ValuingElectronicMusic/network-analysis

Conflicts:
.gitignore