Package: crobat
crobatCrypocurrency Order Book Analysis ToolExplore the docs »View Demo·Report Bug·Request FeatureTable of ContentsAbout the ProjectGetting StartedPrerequisitesInstallationUsageRoadmapContributingLicenseContactAcknowledgementsAbout The ProjectThis project is an extension of my thesis,A Study of CUSUM Statistics on Bitcoin Transactions, where I was tasked with implementing CUSUM statistic processes to identify price actions periods in bitcoin markets. After developing a tool for market orders, the natural extension was to find relationships from activities in the limit order book. I started developing this tool to record instances of the limit order book in order to record Limit Order insertions (LO), cancellations (CO), and Market Orders (MO).As the project grew I wanted to make a tool that could be used by academics looking to apply and develop market microstructure models in live markets. As a result, the styles in which the limit orderbook and orderbook events are recorded are being developed in accordance to the conventions presented in recent market microstructure papers correspond to the following papers:Huang W., Lehalle C.A. and Rosenbaum M. - Simulating and analyzing order book data:The queue-reactive modelCont R., Stoikov S. and Talreja R. - A stochastic model for order book dynamicsCont R., Kukanov A. and Stoikov S. - The price impact of order book eventsCartea. A, Jaimungal S. and Wang Y. - Spoofing and Price Manipulation in Order Driven MarketsSilyantev, E. - Order flow analysis of cryptocurrency marketsThe last paper, 5, shows a working model implementing Order Flow Imbalance (OFI) and Trade Flow Imbalance to BTC-USD trades was done byEd Silyantev. He developed a tool to assess OFI and TFI of XBT-USD pair.Getting StartedTo get a local copy up and running follow these simple steps.PrerequisitesYou can userequirements.txtto see what is necessary but they are also listed below:from standard python library:asyncio, time, datetime, sys, bisectrequisite modules:copra,  pandas, numpyInstallationGiven that this is still very much a work in progress, it may make more sense to fork the project, or download the project as a compressed folder, and buildCSV_out_test.pywith your preferred settings.Note: depending on the popularity of the asset and the computational power of your PC, you may run into errors arising from the computer not being able to keep up with the market (especially BTC-USD). I would suggest experimenting with an unpopular pair, (e.g.,  XRP-USD), or a crypto-crypto pair (e.g., XRP-BTC), and timing your queries outside of NYSE, and London Stock Exchange trading hours as they tend to have less activity.however if you want an easy installation:pip3 install crobatUsageSince this is an orderbook recorder my use until now has been to record the orderbook. However there are accessors in theLOB_funcs.pyfile, under in thehistoryclass. In the /test folder there is a small usecase if you would like to see it but documentation is pending.For now we only have the full orderbook, with no regard for ticksize, and we call thatrecorder_full.py.We change thesettingsvariable in theCSV_out_test.pyfile that has arguments for:ParameterFunction ArgTypeDescriptionRecording Durationdurationintrecording time in secondsPosition Rangeposition_rangeintordinal distance from the best bid(ask)Currency Paircurrency_pairstrList of currency pairs supported by CoinbaseWhen you are ready, you can start the build. When it finishes you should get a messageConnection ClosedfromCoPrA. And the files for the limit orderbook for each side should be created with a timestamp:FilenamesidedescriptionL2_orderbook_events_askYYYY-MM-DDTHH:MM:SS.ffffffaskTime series of order book events on the ask sideL2_orderbook_events_bidYYYY-MM-DDTHH:MM:SS.ffffffbidTime series of order book events on the bid sideL2_orderbook_events_signedYYYY-MM-DDTHH:MM:SS.ffffffbothTime series of order book events on both sides, - sign for bid, +sign for askL2_orderbook_ask_volmYYYY-MM-DDTHH:MM:SS.ffffffaskTime series of the volume snapshots of order book on the ask sideL2_orderbook_bid_volmYYYY-MM-DDTHH:MM:SS.ffffffbidTime series of the volume snapshots of order book on the bid sideL2_orderbook_signed_volmYYYY-MM-DDTHH:MM:SS.ffffffbothTime series of the volume  snapshots of the signed order book, - for bid, + for askL2_orderbook_ask_volmYYYY-MM-DDTHH:MM:SS.ffffffaskTime series of the price snapshots of order book on the ask sideL2_orderbook_bid_volmYYYY-MM-DDTHH:MM:SS.ffffffbidTime series of the price snapshots of order book on the bid sideL2_orderbook_signed_volmYYYY-MM-DDTHH:MM:SS.ffffffbothTime series of the price snapshots of the signed order book, - for bid, + for askUnderstanding The Raw Order Book DataThe coinbase exchange operates using the double auction model, theCoinbase Pro API, and by extension theCoPrA APImakes it realitively easy to get still images of an instance of the orderbook assnapshotsand it sends updates in real time of the volume at a particular price level asl2_updatemessages. If you would like to know more, the cited papers do a great job introducing the double auction model for the purposes of defining the types of orders, and how they record events and make sense of them.Orderbook SnapshotsBelow there is a graph of the snapshot where bids (green) show open limit orders to buy the 1 unit of the cryptocurrency below $7085.930, and asks (red) show open limit orders to buy 1 unit above $7085.930. The x-axis shows the price points, and the y-axis is the aggregate size at the price level. Note that  the signed orderbook calls volume on the bid side negative.Early and current works relied on exchanges and private data providers (e.g.,NASDAQ - BookViewer,LOBSTER) to provide reconstructions of orderbooks. Earlier works were limited to taking snapshots and inferring the possible sequence of orderbook events between states. Coinbase and by extension crobat update the levels on the instance of a update message from the exchange so there is no guess as to what happened between states of the order book. The current format of the orderbook snapshot is not aggregated. The format of the orderbook snapshot for a single side is shown belowTimestamp123...position rangeYYYY-MM-DDTHH:MM:SS.fffffftotal BTC at position 1total BTC at position 2total BTC at position 3...total BTC at position rangeThe associated price quote (price quote (USD per XTC))snapshot is also generated, to make generation of market depth feasible.Timestamp123...position rangeYYYY-MM-DDTHH:MM:SS.ffffffprice quote  at position 1price quote at position 2price quote at position 3...price quote at  position rangeThe signed orderbook takes a different approach to position labelling so please keep that in mind. (note: I should  shift the position index to start at 1, for singe side order book snapshot time series). The signed orderbook snapshot is generated in a similar fashion with a volume, and price at each position. However, it uses the convention established in [3] for the signed order book. where positions on the bid are negative, with negative volume (XTC). I'll show the default setting that displays the 5 best bids and asks on each side.Timestamp-5-4-3-2-112345YYYY-MM-DDTHH:MM:SS.fffffftotal XTC at the 5thbest bidtotal XTC at the 4thbest bidtotal XTC at the 3rdbest bidtotal XTC at the 2ndbest bidtotal XTC at the best bidtotal XTC at the best asktotal XTC at the best 2ndasktotal XTC at the 3rdbest asktotal XTC at the 4thbest asktotal XTC at the 5thbest askSimilar to the single side implementation, there is an associated price quote  (e.g., USD per XTC) snapshot generated at each timepoint. The default format is given below:Timestamp-5-4-3-2-112345YYYY-MM-DDTHH:MM:SS.ffffffprice quote at the 5thbest bidprice quote  at the 4thbest bidprice quote at the 3rdbest bidprice quote at the 2ndbest bidprice quote at the best bidprice quote at the best askprice quote at the best 2ndaskprice quote at the 3rdbest askprice quote at the 4thbest askprice quote at the 5thbest askEvent RecordingsEvent recording are a timeseries of MO, LO, CO's as afforded from thel2_updatemessages which are used to update the price, volume pair size at each price level. The format of the Event recorder is as follows:Timestamporder typeprice levelevent sizepositionmid pricebid-ask spreadYYYY-MM-DDTHH:MM:SS.ffffffMO, LO, COprice level in quote currencyevent size in base currencyposition(best-ask + best-bid)/2best-ask - best-bid rangeSigned event recordings follow the convention fromThe Price impact of Orderbook events, where positive order flow is due to MO's on the buy side, CO on the sell side, and LO on the buy side. Conversely,  negative order flow is due to MO's on the sell side, CO on the buy side, and LO on the buy side. The format is similar to the single side order book events timeseries, but the order volume is signed based on the aforementioned construction.Timestamporder typeprice levelevent sizepositionsidemid pricebid-ask spreadYYYY-MM-DDTHH:MM:SS.ffffffMO, LO, COprice level in quote currencyevent size in base currencysigned position(- for bids, + for asks)buy/sell(best-ask + best-bid)/2best-ask - best-bid rangeRoadmap####Features that need to be developed in order of priority:fixed tick orderbook snapshots and event recordingmarket depth recording in both base and quote currencies.Acessor functions(documentation pending)modernizing/optimizing iteration and classes(replaced sort instances with insert, and a little bit of logic)Finding a way to call the classes outside of the AsyncIO or WebSocket Loop (help me figure this one out!)See theopen issuesfor a list of proposed features (and known issues).ContributingAny contributions you make are greatly appreciated. I am not much of a computer scientist so suggestions and feedback will improve this project for everyone and make me a more capable developer for future projects.Fork the ProjectCreate your Feature Branch (git checkout -b feature/AmazingFeature)Commit your Changes (git commit -m 'Add some AmazingFeature')Push to the Branch (git push origin feature/AmazingFeature)Open a Pull RequestLicenseDistributed under the GNU GPLv3 License. SeeLICENSEfor more information.ContactIvan E. Perez -@IvanEPerez-perez.ivan.e@gmail.comProject Link:https://github.com/orderbooktools/crobatAcknowledgementsTony PodlaskiOlympia Hadjilidis