Merge pull request #2 from annajordanous/master

tidying up first pull from fork at annajordanous
Merge pull request #3 from annajordanous/master

Code to collect user data from SoundCloud for analysis. Tested for small samples (n=100)
Merge pull request #4 from annajordanous/master

updating doc on how to run code
Update README.md
Merge pull request #5 from annajordanous/master

housekeeping
Merge pull request #6 from annajordanous/master

housekeeping
Merge pull request #7 from annajordanous/master

minor bug/display fixes, adding generated graphs
Code for adding data to sqlite database

This is supposed to avoid the bugs that were coming up with SoundCloud
objects missing particular attributes: it adds 'NULL' if an attribute
is missing. But it's also supposed to reduce headaches in another way,
because it generalises the code for adding rows to tables in the
database. You need to create two strings per table, one used for the
SQL statement that creates the table, and one which contains all the
attributes that will be put into columns in the table (this will be the
same as the previous string, minus all the capital letters! I guess I
could have written a regex to convert one to the other, but… it's late).
Removed need for two strings per table in add_data.py

I couldn't stand thinking about all that redundancy. So now there's a
simple function called att_string() that turns the string containing
the bit of SQL that defines the columns in the table into a string that
defines which attributes we want to get from the SoundCloud data
objects and put into columns in the SQLite table. Fewer potential
mistakes!
Expanded into potential replacement for chunks of getSoundCloudData.py

Full comments added, plus a (hopefully) faster version of the function
for adding data to tables. Strings for creating (and populating) the
tables we'll need have now been added, based on what's in
getSoundCloudData.py. I hope that some big chunks of
getSoundCloudData.py will be possible to remove now, replacing them
with calls to functions (and the aforementioned strings) here. Having
all this stuff in a separate module may help to avoid bugs.
Small change to comments

Something else to fix soon.
Now with primary keys!

This required a slightly more complicated function for turning the
table-creator strings into strings containing names of
attributes/columns only. The insert_data function and its loop-based
equivalent now use the string of attribute/column names to specify
exactly which columns data is being inserted into (was able to get away
without this before certain columns were designated as primary keys).
Improved comments in add_data.py

Comments in add_data.py now updated.
New genres & tags script, add_data.py tidied

Added a script that will generate the analysis of genres & tags
presented in the genres2.doc file now in the data folder of our
Dropbox. Also tidied up add_data.py so that the table-creator strings
are packaged up in a dict and create_table and insert_data functions
can now be called with cursor, data, and table names only (the latter
of which they will look up in that dict).
New module for analysing genres & tags

Bit complicated to explain what this is supposed to do - and I don't
have time right now! See test() function in genre_relationships.py for
example.
Comments for genre_relationships.py

Comments now explain what the program does, what needs doing to it, and
what it's for.
Improved comments for genre_relationships.py
genre_relationships.py in fully working order

This now stores all its output in an sqlite database (usable for
further network analysis etc). Instead of requiring the NLTK, now
includes the stopwords from it as a pickled list. Changes to
add_data.py so that it can now do all the database writing to both
sqlite databases.
ongoing work on comments_corp.py
comments_corp.py now in fully working order

comments_corp.py now produces a table in the _deriv database that can
be queried to assemble corpora for comparative analysis. Corpus text
(lightly preprocessed) is in the last column; first column gives the id
from which the original data can be found; other columns contain
metadata variables to refer to in SQL queries when automatically
assembling corpora from the text. See comments at start of file for
explanation.