Package: markets-insights
Markets Insights 📝This package fetches and processes capital markets data from NSE (National Stock Exchange, India). Following data can be retrievedIndex (Nifty, Bank Nifty, NiftyIT)StocksDerivatives (Futures and Options)Support for additional markets and instruments can be added externallyThe package can perform technical functions on price of Index and Stocks. Following indicators are supported.Simple Moving Averages (SMA)Relative Strength Index (RSI)Stochastic RSIBollinger Bands (with standard deviations)The calculation pipeline is quite extensible and more idicators can be added externally.Getting Started 🚀Installation!pipinstallmarkets_insightsfrommarkets_insights.core.environmentimportEnvironmentEnvironment.setup(cache_data_base_path='./cache-data')Get Index data for date rangefrommarkets_insights.datareaderimportdata_readerimportdatetimereader=data_reader.NseIndicesReader()daterange_reader=data_reader.DateRangeDataReader(reader)from_date=datetime.date(2023,1,1)to_date=datetime.date(2023,12,31)result=daterange_reader.read(data_reader.DateRangeCriteria(from_date,to_date))result.head(3).to_markdown()OutputIdentifierIndex DateOpenHighLowClosePoints ChangeChange(%)VolumeTurnoverP/EP/BDiv YieldDate0Nifty 5002-01-202318131.718215.218086.518197.592.150.512.56074e+081.17931e+1121.94.271.262023-01-02 00:00:001Nifty Next 5002-01-202342321.142409.842117.742248.460.750.141.34929e+084.11122e+1025.674.791.762023-01-02 00:00:002Nifty 10002-01-202318290.218350.918237.518334.375.550.413.91003e+081.59043e+1122.754.411.32023-01-02 00:00:00Get daily, monthly and annually aggregrated dataIn this example we will use HistoricalDataProcessor class to get data between a date range. HistoricalDataProcessor will also do monthly and annual aggregation of data.# import classes & setup# import classes & setupfrommarkets_insights.dataprocess.data_processorimportHistoricalDataProcessorfrommarkets_insights.datareader.data_readerimportNseIndicesReader,DateRangeCriteriahistDataProcessor=HistoricalDataProcessor()# Fetch and process the datayear_start=datetime.date(2023,12,1)year_end=datetime.date(2023,12,31)result=histDataProcessor.process(NseIndicesReader(),DateRangeCriteria(year_start,year_end))Displaying the outputWe will callget_monthly_data()method to get monthly aggregated data. We can callget_daily_data()forDailyandget_annual_data()forAnnualdata.frommarkets_insights.core.column_definitionimportBaseColumnsresult.get_monthly_data().sort_values(BaseColumns.Date).head(3)OutputIdentifierDateMonthVolumeTurnoverCloseHighLowOpen0INDIA VIX2023-01-31 00:00:002023-010016.8819.3911.642514.86751012NIFTY100 ENHANCED ESG2023-01-31 00:00:002023-018.12952e+094.99243e+123352.63490.753352.63472.29228NIFTY ALPHA QUALITY VALUE LOW-VOLATILITY 302023-01-31 00:00:002023-012.32482e+091.3303e+1211286.111599.411214.511456.8Calculating RSI using CalculationPipelineBelow example demonstrates calculating RSI using the calculation pipeline.# import classes & setup optionsimportdatetimefrommarkets_insights.datareader.data_readerimportBhavCopyReader,DateRangeCriteriafrommarkets_insights.dataprocess.data_processorimportHistoricalDataProcessor,MultiDataCalculationPipelines,CalculationPipelineBuilder,HistoricalDataProcessOptionsfrommarkets_insights.calculations.baseimportDatePartsCalculationWorkerreader=BhavCopyReader()options=HistoricalDataProcessOptions()options.include_monthly_data=Falseoptions.include_annual_data=FalsehistDataProcessor=HistoricalDataProcessor(options)# Fetch the datayear_start=datetime.date(2023,1,1)to_date=datetime.date(2023,12,31)result=histDataProcessor.process(reader,DateRangeCriteria(year_start,to_date))# Prepare calculation pipelinepipelines=MultiDataCalculationPipelines()pipelines.set_item('rsi',CalculationPipelineBuilder.create_rsi_calculation_pipeline())histDataProcessor.set_calculation_pipelines(pipelines)# Run the pipelinehistDataProcessor.run_calculation_pipelines()Displaying the outputFor displaying the relevant columns in the output we will use column name constants fromBaseColumns&CalculatedColumnsclasses.frommarkets_insights.core.column_definitionimportBaseColumns,CalculatedColumnsresult.get_daily_data().sort_values(BaseColumns.Date).tail(3)[[BaseColumns.Identifier,BaseColumns.Date,BaseColumns.Close,CalculatedColumns.RelativeStrengthIndex]]OutputIdentifierDateCloseRsi336063RKFORGE2023-12-29 00:00:00725.746.9257329710RBL2023-12-29 00:00:00852.9554.8479446931ZYDUSWELL2023-12-29 00:00:001681.172.0492A real use case: Understand the affect of RSI on priceIn this use case, we understand the affect of RSI on the price of equity/stock.Preparing the dataWe perform below steps to prepare our analysis dataCalculate RSI for each day for all the stocks.Add a flag for whenever the RSI crosses the control limits (eg: above 75 and below 30)Calculate the highest and lowest price change in the next 1, 3, 5, 7 & 10 trading sessions.Find the median for highest price change and lowest price change whenever the RSI crosses the control limits.# import classesfrommarkets_insights.datareader.data_readerimportBhavCopyReader,DateRangeCriteria# Fetch the datayear_start=datetime.date(2023,1,1)to_date=datetime.date(2023,12,31)result=histDataProcessor.process(BhavCopyReader(),DateRangeCriteria(year_start,to_date))# prepare calculation pipelineperiods=[1,7,15,30,45]pipelines=MultiDataCalculationPipelines()pipelines.set_item('rsi',CalculationPipelineBuilder.create_rsi_calculation_pipeline(crossing_above_flag_value=75,crossing_below_flag_value=30,window=14))pipelines.set_item('stoch_rsi',CalculationPipelineBuilder.create_stoch_rsi_calculation_pipeline(crossing_above_flag_value=80,crossing_below_flag_value=20,window=14))pipelines.set_item('foward_looking_fall',CalculationPipelineBuilder.create_forward_looking_price_fall_pipeline(periods))pipelines.set_item('foward_looking_rise',CalculationPipelineBuilder.create_forward_looking_price_rise_pipeline(periods))histDataProcessor.set_calculation_pipelines(pipelines=pipelines)# run the pipeline and show resultshistDataProcessor.run_calculation_pipelines()daily_data=result.get_daily_data()# Import constants so its easier to refer to column namesfrommarkets_insights.core.column_definitionimportBaseColumns,CalculatedColumns# get names of fwd looking price column names. Since, these column names are auto-generated there no constants for themfwd_looking_price_fall_cols,fwd_looking_price_rise_cols=[xforxindaily_data.columnsif'HighestPercFallInNext'inx],\[xforxindaily_data.columnsif'HighestPercRiseInNext'inx]Show the median price change % for highest price fall whenever the RSI crosses abovedaily_data[(daily_data[CalculatedColumns.RsiCrossedAbove])][fwd_looking_price_fall_cols].median()OutputHighestPercFallInNext1Days     1.418923
HighestPercFallInNext7Days     3.771446
HighestPercFallInNext15Days    4.777241
HighestPercFallInNext30Days    6.055861
HighestPercFallInNext45Days    6.785467
dtype: float64Show the median price change % for highest price rise whenever the RSI crosses belowdaily_data[(daily_data[CalculatedColumns.RsiCrossedBelow])][fwd_looking_price_rise_cols].median()OutputHighestPercRiseInNext1Days      3.875000
HighestPercRiseInNext7Days      7.589087
HighestPercRiseInNext15Days     9.756772
HighestPercRiseInNext30Days    13.255317
HighestPercRiseInNext45Days    16.292135
dtype: float64Performing Arithmetic on Readersfrommarkets_insights.core.coreimportInstrumentTypeFilterfrommarkets_insights.datareader.data_readerimportBhavCopyReader,NseDerivatiesReader,NseIndicesReaderfrommarkets_insights.core.coreimportIdentifierFilterfor_date=datetime.date(2023,12,5)indices_reader=NseIndicesReader()vix_reader=NseIndicesReader().set_filter(IdentifierFilter("India VIX"))op_reader=indices_reader/vix_readerdata=op_reader.read(ForDateCriteria(for_date)).query(str(IdentifierFilter("Nifty 50 / India VIX")))data.head(3)OutputExtending the Framework: Creating a DataReaderIn this example we will create a new data reader to read data for Nasdaq listed equities. We will useyfinancepython library for this.Import classesfrommarkets_insights.datareader.data_readerimportDateRangeDataReaderfrommarkets_insights.core.coreimportInstrumentationfrommarkets_insights.core.column_definitionimportBaseColumnsimportyfinanceasyfimportpandasCreate reader classWe will create a class that extends the base reader. yfinance library can read data for a range. So, we will extendDateRangeDataReaderclass. With yfinance library, we have to specify which equity/tickers we want to download. For the sake of this example, we will download for top 7 companies of Nasdaq.classNasdaqDataReader(DateRangeDataReader):def__init__(self,tickers:list=['AAPL','GOOGL','MSFT','AMZN','META','TSLA','NVDA']):super().__init__(reader=None)self.tickers=tickersself.name="NasdaqDataReader"@Instrumentation.trace(name="NasdaqDataReader.read")defread(self,criteria:DateRangeCriteria):df_list=list()fortickerinself.tickers:data=yf.download(ticker,group_by="Ticker",start=criteria.from_date,end=criteria.to_date)data['ticker']=tickerdf_list.append(data)# combine all dataframes into a single dataframedf=pandas.concat(df_list)final_data=df.reset_index().rename(columns=self.get_column_name_mappings())final_data[BaseColumns.Date]=pandas.to_datetime(final_data[BaseColumns.Date])returnfinal_datadefget_column_name_mappings(self):return{'ticker':BaseColumns.Identifier,'OPEN':BaseColumns.Open,'HIGH':BaseColumns.High,'LOW':BaseColumns.Low,'CLOSE':BaseColumns.Close}Notice here we are renaming the columns to standard column names so that the calculation pipeline can read them properly.Running the calculation pipelineThe calculation pipeline will not be different except we will pass NasdaqDataReader instance.# import classes & setup optionsimportdatetimefrommarkets_insights.dataprocess.data_processorimportHistoricalDataProcessor,MultiDataCalculationPipelines,CalculationPipelineBuilder,HistoricalDataProcessOptions,DateRangeCriteriafrommarkets_insights.calculations.baseimportDatePartsCalculationWorkerreader=NasdaqDataReader()options=HistoricalDataProcessOptions()options.include_monthly_data=Falseoptions.include_annual_data=FalsehistDataProcessor=HistoricalDataProcessor(options)# Fetch the datayear_start=datetime.date(2023,1,1)to_date=datetime.date(2023,12,31)result=histDataProcessor.process(reader,DateRangeCriteria(year_start,to_date))# Prepare calculation pipelinepipelines=MultiDataCalculationPipelines()pipelines.set_item('date_parts',CalculationPipelineBuilder.create_pipeline_for_worker(DatePartsCalculationWorker()))pipelines.set_item('rsi',CalculationPipelineBuilder.create_rsi_calculation_pipeline())histDataProcessor.set_calculation_pipelines(pipelines)# Run the pipelinehistDataProcessor.run_calculation_pipelines()Here's the code to display resultsfrommarkets_insights.core.column_definitionimportCalculatedColumnsresult.get_daily_data()\.sort_values([BaseColumns.Date,BaseColumns.Identifier])[[BaseColumns.Identifier,BaseColumns.Date,BaseColumns.Close,CalculatedColumns.RelativeStrengthIndex]]\.tail(5)OutputIdentifierDateCloseRsi248AAPL2023-12-28 00:00:00193.5854.4815497AMZN2023-12-28 00:00:00153.3863.9387746GOOGL2023-12-28 00:00:00140.2361.585995META2023-12-28 00:00:00358.3270.23771244MSFT2023-12-28 00:00:00375.2856.9091493NVDA2023-12-28 00:00:00495.2258.3051742TSLA2023-12-28 00:00:00253.1855.9788Extending the Framework: Creating a CalculationWorkerIn this example, we will create a CalculationWorker to calcualte the Fibonacci Retracement level for any equity or index. Finbonacci Retracement levels are based on a time window and a level (26.3%, 50% etc). So, these will become input to our CalculationWorker. Lets call this worker asFibnocciRetracementCalculationWorkerImplement the worker class. The important aspect here is to override theadd_calculated_columns()method## import modulesfrommarkets_insights.calculations.baseimportCalculationWorkerfrommarkets_insights.core.coreimportInstrumentationfrommarkets_insights.calculations.baseimportBaseColumnsimportpandasclassFibonacciRetracementCalculationWorker(CalculationWorker):def__init__(self,time_window:int,level_perct:float):self._time_window=time_windowself._level=level_perct/100self._column_name='Fbr'+str(level_perct)@Instrumentation.trace(name="FibnocciRetracementCalculationWorker")defadd_calculated_columns(self,data:pandas.DataFrame):identifier_grouped_data:pandas.DataFrame=data.groupby(BaseColumns.Identifier)#Since, our dataframe may contain data for multiple symbols, we need to first group them by Identifierdata[self._column_name]=identifier_grouped_data[BaseColumns.Close].transform(lambdax:x.rolling(self._time_window).max()-((x.rolling(self._time_window).max()-x.rolling(self._time_window).min())*self._level))Create pipline with the FibnocciRetracementCalculationWorker and runNow, that our worker is created let us use it in a calculation pipeline. We can use it with any instrument (index, stock) that are supported. Event for the Nasdaq instruments that were supported in earlier examples. For this example, let us take NSE Indexes.# Create pipline with the FibnocciRetracementCalculationWorker and runfrommarkets_insights.datareader.data_readerimportNseIndicesReader,DateRangeCriteriafrommarkets_insights.dataprocess.data_processorimportHistoricalDataProcessor,HistoricalDataProcessOptions,\MultiDataCalculationPipelines,CalculationPipelinehistDataProcessor=HistoricalDataProcessor(HistoricalDataProcessOptions(include_monthly_data=False,include_annual_data=False))# Fetch the dataresult=histDataProcessor.process(NseIndicesReader(),DateRangeCriteria(datetime.date(2023,12,1),datetime.date(2023,12,31)))# Prepare calculation pipelinefbr50_worker=FibonacciRetracementCalculationWorker(time_window=7,level_perct=50)pipelines=MultiDataCalculationPipelines()histDataProcessor.set_calculation_pipelines(CalculationPipeline(workers=[fbr50_worker]))# Run the pipeline and get datahistDataProcessor.run_calculation_pipelines()Display the results.Since our time window was 15 days. So, the calculation result for first 14 days will not be available. We will look at the last 10 records withtail(5)result.get_daily_data()[[BaseColumns.Identifier,BaseColumns.Date,BaseColumns.Close,fbr50_worker._column_name]].tail(5)OutputIdentifierDateCloseFbr502141NIFTY COMPOSITE G-SEC INDEX2023-12-29 00:00:002602.32599.482142NIFTY 10 YR BENCHMARK G-SEC2023-12-29 00:00:002232.792230.242143NIFTY MIDCAP SELECT2023-12-29 00:00:0010397.510209.12144NIFTY ALPHA LOW-VOLATILITY 302023-12-29 00:00:0023373.2228862145NIFTY50 USD2023-12-29 00:00:009048.98941.77