Package: cot-reports
Table of Contentcot_reportsInstallationRequirementsUsageIntroduction to the COT reportsData releaseCOT report typesFormat of the reportsClassification methodology Legacy reportsData aggregationUse cases of the COT reportsSourcesRoadmapLicensecot_reportscot_reportsis a Python library for fetching the Commitments of Trader reports of the Commodity Futures Trading Commission (CFTC). The following Commitments of Trader reports are supported: Legacy Futures-only, Legacy Futures-and-Options Combined, Supplemental Futures-and-Options Combined, Disaggregated Futures-only, Disaggregated Futures-and-Options Combined, Traders in Financial Futures (TFF) Futures-only and Traders in Financial Futures (TFF) Futures-and-Options Combined.Please note: The functionality of the libraries' functions may interrupt in case any changes in the publication occur or in case the website is not available.InstallationUse the package managerpipto install cot_reportspipinstallcot_reportsOr install the library through the author's Github repositorypipinstallgit+https://github.com/NDelventhal/cot_reportsRequirementsThe following libraries are required:pandasrequestsbeautifulsoup4These libraries can be installed via the package managerpip.pipinstallpandasrequestsbeautifulsoup4UsageAvailable report types with the corresponding 'cot_report_type' name used in the library:report'cot_report_type'Legacy Report Futures-only'legacy_fut'Legacy Report Futures-and-Options Combined'legacy_futopt'Supplemental Report Futures-and-Options combined  (short format)'supplemental_futopt'Disaggregated Report Futures-only'disaggregated_fut'Disaggregated Report Futures-and-Options Combined'disaggregated_futopt'Traders in Financial Futures (TFF)'traders_in_financial_futures_fut'Traders in Financial Futures (TFF) Futures-and-Options Combined'traders_in_financial_futures_futopt'Usage examples:cot_hist()downloads the compressed bulk files for the specified report (cot_report_type) starting  from 1986 (depending on the report type) up to 2016 and returns the data in a dataframe.cot_year()downloads historical single year data for the specified report (cot_report_type) and returns the data in a dataframe (cot_report_type). If the current year is specified, the latest published data is fetched.cot_all()downloads the complete available data, including the latest, of the specified report and returns the data in a dataframe (cot_report_type).cot_all_reports()downloads all available historical information of the available COT reports and returns the data as seven dataframes.importcot_reportsascot# Example: cot_hist()df=cot.cot_hist(cot_report_type='traders_in_financial_futures_futopt')# cot_hist() downloads the historical bulk file for the specified report type, in this example the Traders in Financial Futures Futures-and-Options Combined report. Returns the data as dataframe.# Example: cot_year()df=cot.cot_year(year=2020,cot_report_type='traders_in_financial_futures_fut')# cot_year() downloads the single year file of the specified report type and year. Returns the data as dataframe.# Example for collecting data of a few years, here from 2017 to 2020, of a specified report:df=pd.DataFrame()begin_year=2017end_year=2020foriinrange(begin_year,end_year+1):single_year=pd.DataFrame(cot.cot_year(i,cot_report_type='legacy_futopt'))df=df.append(single_year,ignore_index=True)# Example: cot_all()df=cot.cot_all(cot_report_type='legacy_fut')# cot_all() downloads the historical bulk file and all remaining single year files of the specified report type.  Returns the data as dataframe.Introduction to the COT reportsTo promote its goals of integrity, resilience and vibrancy of the U.S. derivatives markets through regulation, the U.S. Commodity Futures Trading Commission (CFTC) relies on collected data to conduct its functions. The CFTC's functions:market oversightmonitoring for liquidity and systemic riskoversight of market participantsregulatory complianceenforcement of the Commodity Exchange Act (CEA)In order to conduct these functions, data is seen as a critical asset to the CFTC, which states to have been one of the first federal agencies who implemented a data management organization in 2011. The CFTC is not solely collecting but also publishing a variety of reports on the derivative markets under its jurisdiction. These reports include the Commitment of Traders (COT) reports.Through the publication of the COT reports the CFTC claims 'to help the public to understand market dynamics.'The COT reports provide a breakdown of each Tuesday’s open interest for futures and options on futures markets in which 20 or more traders hold positions equal to or above the reporting levels established by the CFTC.The data is reported to the CFTC by firms, such as:futures commission merchants (FCMs)clearing membersforeign brokersexchangesThe actual trader classification is based on the predominant business purpose, which is self-reported by traders on the CFTC Form 401. The classification of a trader can vary for different commodities/markets, if the reported business purposes differ. However, the classifications are subject to review and corrections by the CFTC.Data releaseThe data, which is generally released each Friday at 3:30 pm Eastern time, comes with a lag of three days, as the reported data is typically from previous Tuesday (close). According to the commission the firms report to the CFTC on Wednesday morning. Next, the provided information is then subject to corrections and verifications by the CFTC prior to the release on Friday.COT report typesThe types of the COT reports:Legacyphysical contracts & non-physical contractsreportable position classifications:non-commercialcommercial tradersavailable as Futures-only and Futures-and-Options Combined reportsavailable from 1986 and 1995available in long and short formatSupplementalselected agricultural commodity contractsSupplemental reports break down the reportable open interest positions into three trader classificationsreportable position classifications:non-commercialcommercialindex tradersavailable as Futures-and-Options Combined reportavailable from 2006available in short formatWhy was this report established? With the publication of the Supplemental COT report the CFTC has reacted on the rise of index traders. The commission started to publish the Supplemental report starting in 2006 with a further trader categorization of index traders for selected agricultural products to prevent a blending of traditional commercial positions and also of traditional non-commercial position with index funds positions. With the newly introduced classification of index traders misinterpretation related to export activity buying or selling end-user are avoided, by differentiating between traditional and non-traditional hedgers as well as between traditional non-commercial and index funds open interest. In general from the non-commercial category managed funds, pension funds and other institutional investors that are generally seeking exposure to commodity prices as an asset class in an unleveraged and passively-managed manner through standardized commodity indices are drawn into the index trader categorization. From the commercial category entities, whose positions predominantly reflect hedging of OTC transactions through commodity indices, are drawn into the index trader categorization. The Supplemental reports solved a disadvantage with respect to the usefulness of the information and serves the original intent in publishing the COT data better by enhancing the transparency.Disaggregateddata on physical contracts, such as contracts on agricultural products, petroleum products, natural gas products etc.reportable position classifications:Producer/Merchant/Processor/UserSwap DealersManaged MoneyOther Reportableavailable as Futures-only and Futures-and-Options Combined reportsavailable from 2006available in long and short formatTraders in Financial Futuresdata based on non-physical contractsreportable position classifications:Dealer/IntermediaryAsset Manager/InstitutionalLeveraged FundsOther Reportableavailable as Futures-only and Futures-and-Options Combined reportsavailable from 2006available in long formatFormat of the reportsThe COT reports are either available in the short format, the long format or both - as specified above. Within the long format, aside from the additional listing of the concentrations of positions held by the largest four and largest eight traders, the data is grouped by crop year, where appropriate. Except, where not available (cot report type: Supplemental), the library accesses the long format.Data aggregationAggregations occur in the reports not only in the classification of the trader, but also by merging future positions with differing expiration dates. Due to this handling continuous sentiment data on a product, referred to as market by the CFTC and within the COT reports, can be derived. As the data is highly aggregated, any results derived from the data should be interpreted with caution.Classification Methodology Legacy reportsNot only does the Legacy Futures-only COT report offer the greatest available historical data dating back to 1986, the Legacy reports also contain the most markets in comparison to the other COT report types. Unlike the others, the Legacy reports contain data on physical delivery contracts and non-physical delivery (financial) contracts. While the Supplemental and the Disaggregated reports are limited to physical delivery contracts, the Traders in Financial Futures reports are limited to non-physical financial contracts.Classification Methodology: Legacy reportsThe positions of the market participants in the Legacy reports are broken down into:Commercialsmarket participants who use futures contracts for hedging as defined in CFTC Regulation 1.3, 17 CFR 1.3(z) and which hold positions equal to or above the reporting levels established by the CFTCCFTC Form 40 declaration: the trader is engaged in business activities, which can be hedged by the use of futures (or options)Example: A gold mining company, such as Goldcorp or Barrick Gold, would be classified as commercials in gold futures, as well as other merchants, processors or users buying and selling Goldusing descriptions of the other reports I derive the commercials to contain primarily producer/merchant/processor/user and swap dealersNon-commercialsmarket participant whose positions equal or are above the reporting levels of by the CFTC and are not considered as commercialsAs a consequence this group is derived to have speculative interest and is commonly considered as speculatorsusing descriptions of the other reports I derive the non-commercials to contain primarily managed money, asset manager/institutional and leveraged fundsNon-reportablethese market participants hold positions below the reporting levels of the CFTCwhile the to the CFTC reporting firms are not informing the commission on this group, the aggregated non-reportable positions are nevertheless known to the CFTC through its information on the open interest data (and the reported commercials and non-commercials details)Use cases of the COT reportsThere are numerous, partially overlapping, use cases for the COT data of the CFTC.
Here are the ones I can think of:Regulatory/Academic:formulate recommendations to the CFTC, such asenlarging transparencyestablishing new monitoring facilities through Machine Learningadding specific changes to the report to improve detecting misbehaviours of markets participantsanalyse effects of past regulatory changes and regulatory demandsanalyse price manipulations (this may only be identifiable through the CFTC’s confidential trader-level data, which is not accessible, but less aggregated data than publicly available has been provided to a number of researchers in the past)add research/recommendations on limiting speculative bets on commoditiesevaluate if other trading commissions (other countries) should establish a COT like database to combat misbehauviours and as a screening tool for market risksanalyse the usage of leverage and possible worst case scenarios targeting preventions of financial harm to commercials/investorsCentral banks:assess liquidity and systemic riskassess bets on/against a currency, evaluate impact of decisions or costs of defending a thresshold in a currency pair; example: the Swiss National Bank (SNB) defended the 1.2 level in the EUR/CHF pair for years up until early 2015 and artificially stabilized the currency through activities, particularly in the treasuries and currency markets (including other CHF cross pairs). Further, the SNB's decision to not continue the support of the price limit caused a significant shock to the EUR/CHF price due to massive long exposure/herding of speculators and a severe liquidity shortage below the mark of 1.2.Market participants:Producers/Merchants:can monitor the behaviour of their competitors to time actions (hedge positions) or other strategic decisions; example: a farmer might decide on what to grow next season based on the commercial positionSpeculators:evaluate if information is priced in (high vs. low position changes after events with potentially high impact)analyse the behaviour of competitorsdetect the 'mood of the markets' (risk aversion or risk appetite of market participants)derive trading approachesMarket observers:for buy-side analysts researchfor coverage of financial journalistsmonitoring of changes in positions or their values in a comparable metric, once derived, such as the U.S. dollar(Behavioural) Finance:add value to the research of market dynamics and trends/herdinganalyse reactions/changes in positions prior and following market events, such as interest rate decisions or other macroeconomic factors/shockscomparison/understanding of not only changes in price, but also activities in the marketseffects of political events or geopolitical eventsdetecting mispricing through extremes in the positionData vendors:hosting data in a cleaned form (as the data contains somes cleanable parts) for enhanced frictionless accessAcceptance of new technologies through commercial activitiesanalysis of the acceptance of new technologies, through a screening if evidence for increased commercial interest activities can be found; example: commercial activities in Bitcoin futures at the Chicago Mercantile Exchange (CME)SourcesCOT reports data source:Commitments of Traders - Historical CompressedSources of the above information on the CFTC and the COT reports:CFTC About: MissionCFTC Market Reports: Commitments of TradersCFTC Commission Actions in Response to the 'Comprehensive Review of the Commitments of Traders Reporting Program' (June 21, 2006)CFTC FORM 40 STATEMENT OF REPORTING TRADERRoadmapcreation of the predecessor of this library within the seminar course 'Statistical Programming and Open Science Methods (PhD-level)' of the Humboldt University of Berlin (Q1 2020)adding library to The Python Package Index (PyPI)LicenseThis project is licensed under theMIT License.ContactThe author: Niall Delventhal -ni.delventhal@gmail.com