Package: betfairdatabase
betfair-databaseTurns a collection of historical Betfair data into a queryable SQL database.InstallationInstall the package fromPyPI:pipinstallbetfairdatabaseUsageGetting startedIndex the folder holding historical Betfair data to turn it into a database.Use SQL queries to select data.importbetfairdatabaseasbfdbpath_to_data="./my_betfair_data"bfdb.index(path_to_data)# Create an index to convert the folder into a database# Select all greyhound races in Sheffielddataset=bfdb.select(path_to_data,where="eventTypeId='4339' AND eventVenue='Sheffield'")formarketindataset:print(market["marketDataFilePath"],# Path to stream data filemarket["marketCatalogueFilePath"],# Path to market catalogue file)The historical data can be grouped and divided using any subfolder hierarchy, but it must follow this convention:Market catalogue is stored in a JSON file named<market id>.json.Market data file (containing stream data) is stored in the same folder as the market catalogue file. It shares the same basename<market id>and ends with.zip,.gzor.bz2, or it has no extension (uncompressed data).A sample database structure is shown below:my_betfair_data/
├── arbitrary_folder/
    ├── 1.22334455.json  # Market catalogue file
    ├── 1.22334455  # Uncompressed market data file
    ├── 1.55667788.json  # Market catalogue file
    └── 1.55667788.zip  # Compressed market data fileRetrieving dataselect()method accepts the following arguments:database_dir: Main directory of the database initialised withindex().columns: A list of columns (field names) to retrieve. If omitted, all columns are returned. View a list of available columns by callingbetfairdatabase.columns().where: SQL query to execute.limit: Maximum number of results to return. If omitted, all results are returned.return_dict: IfTrue(default), results are returned as a dictionary where keys are column names and values are data. IfFalse, results are returned as tuples containing only data. The second option is faster but makes data harder to work with.Below are several examples of selecting and filtering data:importbetfairdatabaseasbfdbpath_to_data="./my_betfair_data"# Return all market ids and paths to data files in the databasebfdb.select(path_to_data,columns=["marketId","marketDataFilePath"])# Return full market metadata for horse racing win marketsbfdb.select(path_to_data,where="eventTypeId='7' AND marketType='WIN'")# Return full market metadata for a maximum of 100 BSP marketsbfdb.select(path_to_data,where="bspMarket=true",limit=100)# Return a maximum of 250 data file paths for horse and greyhound racingbfdb.select(path_to_data,columns=["marketDataFilePath"],where="eventTypeId IN ('7', '4339') AND marketType='WIN'",limit=250,)Inserting dataDatabase can be updated with new files usinginsertmethod. This is much faster and more efficient than reindexing the whole database on each update. Files are moved by default, but they can also be copied ifcopy=Trueargument is provided.importbetfairdatabaseasbfdbbfdb.insert("./my_betfair_data","./my_capture_dir")Exporting dataDatabase index can be exported to a CSV file with theexport()method. This is useful for debugging, visualising data and post-processing it with external tools.importbetfairdatabaseasbfdbcsv_file=bfdb.export("./my_betfair_data","./my_data_dump")print(csv_file)# Prints: ./my_data_dump/my_betfair_data.csvRemoving missing dataThroughout the course of database's lifetime, indexed files may get removed.clean()method checks for the presence of indexed market data files and removes the missing entries from the index, avoiding the need to reindex the whole database on every single file removal. However, reindexing the database may be the faster option when a large number of files has been removed.importbetfairdatabaseasbfdbbfdb.clean("./my_betfair_data")Object-oriented interfaceAll of the above methods can also be accessed through OOP interface viaBetfairDatabaseclass. This is useful when performing multiple operations on the same database as the database directory needs to be provided only once.frombetfairdatabaseimportBetfairDatabasedb=BetfairDatabase("./my_betfair_data")db.index()db.select()db.insert("./my_capture_dir")db.export()db.clean()Command line interfaceThe package also installs abfdbcommand line app, which provides access to the following methods:bfdbindex"./my_database_dir"# Index a databasebfdbexport"./my_database_dir""./my_db_dump.csv"# Export a databasebfdbinsert"./my_database_dir""./my_captured_data"# Update the databasebfdbclean"./my_database_dir"# Clean the databaseFor more information about the command line interface, run:bfdb--help