Package: deephaven-ib
deephaven-ibAnInteractive Brokersintegration forDeephaven.Interactive Brokersis a very popular brokerage in the quantitative finance world,
with about $200B of customer equity.  Quants and hedge funds often chooseInteractive Brokersbecause of its low trading costs and API that facilitates automated trading.  With low minimum account balances,
it is also an attractive choice for individual investors.Deephavenis the real-time query engine that is the backbone for the quantitative trading of the
world's largest hedge funds, banks, and exchanges.Deephavenmakes working with real-time data easy and
facilitates very concise and easy-to-read code.  WithDeephaven, quants can create new models
and get them into production quickly, traders can monitor the market and their portfolios, and
managers can monitor risk.deephaven-ibcombines the low-cost trading ofInteractive Brokerswith the analytical power and ease of use ofDeephaven Community Coreto yield an open, quantitative
trading platform.  Basically, it provides an open platform for building quantitative trading strategies and
custom analytics.  You can build something simple, like a portfolio monitor, or something complex, like a
fully-automated, multi-strategy quantitative hedge fund.deephaven-ibsupports trading essentially all common
exchange traded products.  These include:StocksMutual FundsOptionsFuturesFutures OptionsIndexesBondsForeign Exchange (Forex or FX)CryptocurrencyContracts for Differences (CFDs)WarrantsCommoditiesWARNING: Automated trading can go horribly wrong very quickly.  Verify your code on a paper trading account before
unleashing trading on an account where money can be lost.  If you think this can not happen to you, readThe Rise and Fall of Knight Capital.
TheSetupsection shows configurations to prevent accidental trade submission.For more details, see:Interactive BrokersDeephavenDeephaven Community CoreFor help withdeephaven-ib:deephaven-ib DocsGitter: A relaxed chat room about all things DeephavenDeephaven Community SlackFor Deephaven how-to guides, see:Deephaven TutorialDeephaven Coummunity Core Documentation.For help withDeephaven:Gitter: A relaxed chat room about all things DeephavenDeephaven Community SlackDeephaven Community Core DiscussionsData Available in DeephavenTheDeephavenquery engine is built around the concept of tables, which are similar to Pandas dataframes.Unlike Pandas dataframes,Deephaventables can dynamically update as new data is streamed in.
As input tables change, theDeephavenquery engine ensures that all queries, no matter how complex,
are kept up-to-date.Once data is converted to aDeephaventable, it can be used in queries with any otherDeephaventables.IB TWS dataData available fromIB Trader Workstation (TWS)can be accessed
asDeephaventables by usingdeephaven-ib.
As data streams in, the tables and queries using them will automatically update.These tables include:Generalerrors: an error logrequests: requests to IBContractscontract_details: details describing contracts of interest.  Automatically populated.contracts_matching: contracts matching query strings provided torequest_contracts_matching.market_rules: market rules indicating the price increment a contract can trade in.  Automatically populated.short_rates: interest rates for shorting securities.  Automatically populated ifdownload_short_rates=True.Accountsaccounts_managed: accounts managed by the TWS session login.  Automatically populated.accounts_family_codes: account family.  Automatically populated.accounts_groups: account groups.  Automatically populated.accounts_allocation_profiles: allocation profiles for accounts.  Automatically populated.accounts_value: account values.  Automatically populated.accounts_overview: overview of account details.  Automatically populated.accounts_summary: account summary.  Automatically populated.accounts_positions: account positions.  Automatically populated.accounts_pnl: account PNL.  Automatically populated.Newsnews_providers: currently subscribed news sources.  Automatically populated.news_bulletins: news bulletins.  Automatically populated.news_articles: the content of news articles requested viarequest_news_article.news_historical: historical news headlines requested viarequest_news_historical.Market Dataticks_price: real-time tick market data of price values requested viarequest_market_data.ticks_size: real-time tick market data of size values requested viarequest_market_data.ticks_string: real-time tick market data of string values requested viarequest_market_data.ticks_efp: real-time tick market data of exchange for physical (EFP) values requested viarequest_market_data.ticks_generic: real-time tick market data of generic floating point values requested viarequest_market_data.ticks_option_computation: real-time tick market data of option computations requested viarequest_market_data.ticks_trade: real-time tick market data of trade prices requested viarequest_tick_data_historicalorrequest_tick_data_realtime.ticks_bid_ask: real-time tick market data of bid and ask prices requested viarequest_tick_data_historicalorrequest_tick_data_realtime.ticks_mid_point: real-time tick market data of mid-point prices requested viarequest_tick_data_historicalorrequest_tick_data_realtime.bars_historical: historical price bars requested viarequest_bars_historical.bars_realtime: real-time price bars requested viarequest_bars_realtime.Order Management System (OMS)orders_submitted: submitted ordersFOR THE THE CLIENT'S ID.  A client ID of 0 contains manually entered orders.  Automatically populated.orders_status: order statuses.  Automatically populated.orders_completed: completed orders.  Automatically populated.orders_exec_details: order execution details.  Automatically populated.orders_exec_commission_report: order execution commission report.  Automatically populated.Most tables include aReceiveTimecolumn.  This column indicates the time the data was received bydeephaven-ib. It does not represent the time the event occurred.Your dataYou may want to combine data from other sources with your IB data.Deephavencan load data from:CSVParquetKafka.See theDeephaven Documentationfor details.Files placed in the./docker/data/directory are visible in the Docker container at/data/.SeeAccess your file system with Docker data volumesfor details.Run deephaven-ibFollow these steps to run aDeephavenplusInteractive Brokerssystem.Windows users need to run the commands in WSL.SetupTo setup and configure the system:Follow theDeephaven Quick Start Guideto getDeephavenrunning.Follow theTWS Installation Instructionsto getIB Trader Workstation (TWS)running.LaunchIB Trader Workstation (TWS).InIB Trader Workstation (TWS), click on the gear in the
upper right corner.InAPI->Settings, make sure:"Enable ActiveX and Socket Clients" is selected."Allow connections from localhost only" is not selected."Read-Only API" is selected if you want to prevent trade submission fromdeephaven-ib.Also, note the "Socket port" value.  It is needed when connectingdeephaven-ib.[For Paper Trading] Log into theInteractive Brokers Web Interface.[For Paper Trading] In theInteractive Brokers Web Interface, navigate toAccount->Settings->Paper Trading Accountand make sure that "Share real-time market data subscriptions with paper trading account?" is set to true.Oncedeephaven-ibis launched (seebelow), accept incoming connections toIB Trader Workstation (TWS).  (May not be required for all sessions.)LaunchThere are multiple ways to launchDeephavenwithdeephaven-ibinstalled.  The launch can either happen via a local installation or via Docker images.For a Docker Compose example, seedocker/release/.(Option 1) Launch pip-installed Deephaven with Docker -- interactiveThe pip-installed Deephaven uses a lightweight Deephaven installation that is installed using pip.  In this case,
the pip-installed Deephaven system is installed in a Docker container.Create a directory for your data and scriptsmkdirdataCreate a directory for your Deephaven IDE configuration and notebooksmkdir`pwd`/.deephavenLaunch the system:# Set jvm_args to the desired JVM memory for Deephavendockerrun-it-vdata:/data-v`pwd`/.deephaven:/storage-p10000:10000ghcr.io/deephaven-examples/deephaven-ibpython3-i-c"from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"Launch theDeephaven IDEby navigating tohttp://localhost:10000/ide/in a browser.(Option 2) Launch pip-installed Deephaven with Docker -- run a scriptThe pip-installed Deephaven uses a lightweight Deephaven installation that is installed using pip.  In this case,
the pip-installed Deephaven system is installed in a Docker container.This is a good option for production scenarios where scripts need to be run and related data needs to be visualized.Create a directory for your data and scriptsmkdirdata# your_script.py must begin with: "from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g']); _server.start()"# Set jvm_args to the desired JVM memory for Deephavencppath/to/your_script.pydata/your_script.pyCreate a directory for your Deephaven IDE configuration and notebooksmkdir`pwd`/.deephavenLaunch the system:# Set jvm_args to the desired JVM memory for Deephavendockerrun-it-vdata:/data-v`pwd`/.deephaven:/storage-p10000:10000ghcr.io/deephaven-examples/deephaven-ibpython3-i/data/your_script.pyLaunch theDeephaven IDEby navigating tohttp://localhost:10000/ide/in a browser.(Option 3) Launch pip-installed Deephaven with a local installation (No Docker) -- interactiveThe pip-installed Deephaven uses a lightweight Deephaven installation that is installed using pip.  In this case,
the pip-installed Deephaven system is installed directly on your local system, without Docker.It is possible to usedeephaven-ibwithout docker, but this is a
new feature and has not been well tested.  To do this:Installibapi:# pip installed version of ibapi is too old.  You must download and install a more recent version.exportIB_VERSION=1016.01
curl-o./api.zip"https://interactivebrokers.github.io/downloads/twsapi_macunix.${IB_VERSION}.zip"unzipapi.zipcd./IBJts/source/pythonclient
python3setup.pyinstallInstall Java 11 and set the appropriateJAVA_HOMEenvironment variable.Installdeephaven-ib:pip3install--upgradepipsetuptoolswheel
pip3installdeephaven-ibLaunch the system:# Set jvm_args to the desired JVM memory for Deephaven# Deephaven IDE configuration and notebooks are stored to ~/.deephavenpython3-i-c"import os; from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g','-Dstorage.path=' + os.path.expanduser('~/.deephaven')]); _server.start()"Launch theDeephaven IDEby navigating tohttp://localhost:10000/ide/in a browser.Usehost="localhost"for the hostname in the examples (Windows WSL useshost="host.docker.internal", since WSL is built on Docker.)(Option 4) Launch pip-installed Deephaven with a local installation (No Docker) -- run a scriptThe pip-installed Deephaven uses a lightweight Deephaven installation that is installed using pip.  In this case,
the pip-installed Deephaven system is installed directly on your local system, without Docker.This is a good option for production scenarios where scripts need to be run and related data needs to be visualized.It is possible to usedeephaven-ibwithout docker, but this is a
new feature and has not been well tested.  To do this:Installibapi:# pip installed version of ibapi is too old.  You must download and install a more recent version.exportIB_VERSION=1016.01
curl-o./api.zip"https://interactivebrokers.github.io/downloads/twsapi_macunix.${IB_VERSION}.zip"unzipapi.zipcd./IBJts/source/pythonclient
python3setup.pyinstallInstall Java 11 and set the appropriateJAVA_HOMEenvironment variable.Installdeephaven-ib:pip3install--upgradepipsetuptoolswheel
pip3installdeephaven-ibLaunch the system and execute a custom script:# your_script.py must begin with: "import os; from deephaven_server import Server; _server = Server(port=10000, jvm_args=['-Xmx4g','-Dstorage.path=' + os.path.expanduser('~/.deephaven')]); _server.start()"# Deephaven IDE configuration and notebooks are stored to ~/.deephaven# Set jvm_args to the desired JVM memory for Deephavenpython3-iyour_script.pyLaunch theDeephaven IDEby navigating tohttp://localhost:10000/ide/in a browser.Usehost=localhostfor the hostname in the examplesUse deephaven-ibTo usedeephaven-ib, you will need to open theDeephaven Web IDEby navigating tohttp://localhost:10000/ide/in your web browser.  The following commands
can be executed in the console.Connect to TWSAlldeephaven-ibsessions need to first create a client for interacting
withIB Trader Workstation (TWS).hostis the computer to connect to.  When usingdeephaven-ibinside
of Docker,hostshould be set tohost.docker.internal.portis the network portIB Trader Workstation (TWS)communicates on.  This value can be found in theIB Trader Workstation (TWS)settings.  By default, production trading uses port 7496, and paper trading uses port 7497.  SeeSetupandTWS Initial Setupfor more details.read_onlyis a boolean value that is used to enable trading.  By defaultread_only=True, preventing trading.  Useread_only=Falseto enable trading.is_fais a boolean value that is used to indicate if an account is a financial advisor (FA) account or a regular acccount.
By usingis_fa=True, FA account configuration details are requested.  By defaultis_fa=False.Ifis_fa=Trueis used on a non-FA account, everything should work fine, but there will be error messages.
Ifis_fa=False(the default) is used on a FA account, FA account configurations will not be populated in tables such asaccounts_groups,accounts_allocation_profiles, andaccounts_aliases.order_id_strategyis the strategy used for obtaining new order ids.  Order id algorithms have tradeoffs in execution time, support for multiple, concurrent sessions, and avoidance of TWS bugs.OrderIdStrategy.RETRY(default) - Request a new order ID from TWS every time one is needed.  Retry if TWS does not respond quickly.  This usually avoids a TWS bug where it does not always respond.OrderIdStrategy.BASIC- Request a new order ID from TWS every time one is needed.  Does not retry, so it may deadlock if TWS does not respond.OrderIdStrategy.INCREMENT- Use the initial order ID sent by TWS and increment the value upon every request.  This is fast, but it may fail for multiple, concurrent sessions connected to TWS.For a read-write session that allows trading:importdeephaven_ibasdhibclient=dhib.IbSessionTws(host="host.docker.internal",port=7497,read_only=False)client.connect()For a read-only session that does not allow trading:importdeephaven_ibasdhibclient=dhib.IbSessionTws(host="host.docker.internal",port=7497,read_only=True)client.connect()For a read-only financial advisor (FA) session that does not allow trading:importdeephaven_ibasdhibclient=dhib.IbSessionTws(host="host.docker.internal",port=7497,read_only=True,is_fa=True)client.connect()Afterclient.connect()is called, TWS requires that the connection be accepted.Get dataIB Trader Workstation (TWS)data is stored in
thedeephaven-ibclient as two dictionaries of tables:tablescontains the tables most users will want.tables_rawcontains rawIB Trader Workstation (TWS)data.As an example, therequeststable, that contains all of the requests made toIB Trader Workstation (TWS),
can be obtained by:requests=client.tables["requests"]To display all of the tables in theDeephaven IDE,
place the tables in the global namespace.  This can most easily be done by:fork,vinclient.tables.items():globals()[k]=vSimilarly, raw tables can be viewed by:fork,vinclient.tables_raw.items():globals()[k]=vA list of available tables can be obtained by:print(client.tables.keys())print(client.tables_raw.keys())Create a contractIn IB, financial contracts include:StocksFXCryptocurrencyIndexesCFDsFuturesOptionsFutures OptionsBondsMutual FundsWarrantsCommoditiesTo create a contract for use indeephaven-ib,
the contract must first be created as anibapi.contract.Contract.  Once the contract is created,
it must be registered withdeephaven-ibbefore it
can be used.Details on creating contracts can be found athttps://interactivebrokers.github.io/tws-api/basic_contracts.html.Registering the contract causes the contract details to appear in thecontracts_detailstable.fromibapi.contractimportContractc=Contract()c.symbol='AAPL'c.secType='STK'c.exchange='SMART'c.currency='USD'rc=client.get_registered_contract(c)print(rc)./examples/example_all_functionality.pyillustrates the creation and registration
of many different types of contracts.Request market dataMarket data can be requested from the client using:request_market_datarequest_bars_historicalrequest_bars_realtimerequest_tick_data_realtimerequest_tick_data_historicalfromibapi.contractimportContractimportdeephaven_ibasdhib# Use delayed market data if you do not have access to real-time# client.set_market_data_type(dhib.MarketDataType.DELAYED)client.set_market_data_type(dhib.MarketDataType.REAL_TIME)c=Contract()c.symbol='AAPL'c.secType='STK'c.exchange='SMART'c.currency='USD'rc=client.get_registered_contract(c)print(rc)client.request_market_data(rc)client.request_tick_data_realtime(rc,dhib.TickDataType.BID_ASK)client.request_tick_data_realtime(rc,dhib.TickDataType.LAST)client.request_tick_data_realtime(rc,dhib.TickDataType.MIDPOINT)./examples/example_all_functionality.pyillustrates requesting
many kinds of market data.Request newsMarket data can be requested from the client using:request_news_historicalrequest_news_articlefromibapi.contractimportContractfromdeephaven.timeimportto_datetimecontract=Contract()contract.symbol="GOOG"contract.secType="STK"contract.currency="USD"contract.exchange="SMART"rc=client.get_registered_contract(contract)print(contract)start=to_datetime("2021-01-01T00:00:00 NY")end=to_datetime("2021-01-10T00:00:00 NY")client.request_news_historical(rc,start=start,end=end)client.request_news_article(provider_code="BRFUPDN",article_id="BRFUPDN$107d53ea")./examples/example_all_functionality.pyillustrates requesting
news data.Request account detailsStandard account details are requested by default.IB Trader Workstation (TWS)does not provide an API for requesting all model codes, sodeephaven-ibcan not subscribe to data for different model codes.  If you need details on non-standard
account / model code combinations, you can use:request_account_pnlrequest_account_overviewrequest_account_positionsOrder managementOrders can be created and canceled using:order_placeorder_cancelorder_cancel_allTo place an order, register a contract withdeephaven-ib,
and create anibapi.order.Ordercontaining details for the order.Details on creating orders can be found athttps://interactivebrokers.github.io/tws-api/orders.html.fromibapi.contractimportContractfromibapi.orderimportOrdercontract=Contract()contract.symbol="GOOG"contract.secType="STK"contract.currency="USD"contract.exchange="SMART"rc=client.get_registered_contract(contract)print(contract)order=Order()order.account="DF4943843"order.action="BUY"order.orderType="LIMIT"order.totalQuantity=1order.lmtPrice=3000order.eTradeOnly=Falseorder.firmQuoteOnly=Falsereq=client.order_place(rc,order)req.cancel()client.order_place(rc,order)client.order_cancel_all()Queries and MathematicsDeephavenhas very powerful query engine that allows mathematics and queries
to be applied to static and real-time data.  The queries can be as simple as filtering data and
as complex as artificial intelligence.The example below computes the real-time price ratio ofDIA(Dow Jones Index) andSPY(S&P 500 Index)
every 5 seconds.For more details, see theDeephaven Coummunity Core Documentation.fromibapi.contractimportContractc1=Contract()c1.symbol='DIA'c1.secType='STK'c1.exchange='SMART'c1.currency='USD'rc1=client.get_registered_contract(c1)print(rc1)c2=Contract()c2.symbol='SPY'c2.secType='STK'c2.exchange='SMART'c2.currency='USD'rc2=client.get_registered_contract(c2)print(rc2)client.set_market_data_type(dhib.MarketDataType.REAL_TIME)client.request_market_data(rc1)client.request_market_data(rc2)client.request_bars_realtime(rc1,bar_type=dhib.BarDataType.MIDPOINT)client.request_bars_realtime(rc2,bar_type=dhib.BarDataType.MIDPOINT)bars_realtime=client.tables["bars_realtime"]bars_dia=bars_realtime.where("Symbol=`DIA`")bars_spy=bars_realtime.where("Symbol=`SPY`")bars_joined=bars_dia.view(["Timestamp","TimestampEnd","Dia=Close"])\.natural_join(bars_spy,on="TimestampEnd",joins="Spy=Close")\.update("Ratio = Dia/Spy")PlottingDeephavenhas very powerful plotting functionality for both static and real-time data.
The example below plots the bid and ask prices ofAAPLfor every tick in the market.For more details, see theDeephaven Coummunity Core Documentation.fromibapi.contractimportContractc=Contract()c.symbol='AAPL'c.secType='STK'c.exchange='SMART'c.currency='USD'rc=client.get_registered_contract(c)print(rc)client.set_market_data_type(dhib.MarketDataType.REAL_TIME)client.request_market_data(rc)client.request_tick_data_realtime(rc,dhib.TickDataType.BID_ASK)ticks_bid_ask=client.tables["ticks_bid_ask"]fromdeephaven.plotimportFigureplot_aapl=Figure().plot_xy("Bid",t=ticks_bid_ask,x="ReceiveTime",y="BidPrice")\.plot_xy("Ask",t=ticks_bid_ask,x="ReceiveTime",y="AskPrice")\.show()Help!Error Tabledeephaven-iblogs allIB Trader Workstation (TWS)errors to theerrorstable.  This table should be monitored when usingdeephaven-ib.errors=client.tables["errors"]Loggingdeephaven-ibandibapiboth use Python'sloggingframework.  By default,ERRORand higher
levels are logged.  More or less logging can be displayed by changing the logging level.To see fewer log messages:importlogginglogging.basicConfig(level=logging.CRITICAL)To see all log messages:importlogginglogging.basicConfig(level=logging.DEBUG)A discussion of available logging levels can be found in thePythonloggingmodule documentation.SupportIf you can not solve your problems through either theerrorstable or through logging, you can try:deephaven-ib API DocumentationInteractive Brokers SupportGitter: A relaxed chat room about all things DeephavenDeephaven Community SlackTakes N positional arguments but M were givenYou may encounter an error that looks like:Takes N positional arguments but M were given.  If you see a problem like this, youribapiversion does not match the version needed bydeephaven-ib.  Theibapiversion in PyPIis ancient and appears to have been abandoned byInteractive Brokers.  CurrentlyInteractive Brokersis deliveringibapieither via theIB Trader Workstation (TWS)download or via git.To check youribapiversion:importibapiprint(ibapi.__version__)If your version is9.x, it is the old version fromPyPI.  To install the requiredibapiversion:# pip installed version of ibapi is too old.  You must download and install a more recent version.exportIB_VERSION=1016.01
curl-o./api.zip"https://interactivebrokers.github.io/downloads/twsapi_macunix.${IB_VERSION}.zip"unzipapi.zipcd./IBJts/source/pythonclient
python3setup.pyinstallNote that theibapiAPI is very unstable.  You likely need the exact version mentioned here fordeephaven-ibto function.  If you want a better installation experience and more flexability with theibapiversion, reach out toInteractive Brokersand let them know that you want them to start publishing the latestibapiversions toPyPI.ExamplesExamples can be found in./examples.