Package: orgpedia-cabsec
Data-package: orgpedia_cabsecPosting data of Ministers of India. The data is obtained by processing posting orders fromCabinet Secretariat's website.To get a quick peek check out thetenures-sample.csvit contains a snapshot of the tenure information of Cabinet Secretariat officers.The tenures information is built by processing orders found on the Cabinet Secretariat's webpage (import/documents). The orders are processed to build higher level concepts of Tenure and an Org chart. To undersand the processing logic please check out theData Processingsection.Accessing the dataAll the data is available in theflow/buildTenure_/outputfolder and it contains the following filestenures.json,tenures.csv: Tenure information in json and csv formatorders.json: Order information in json format.officer_infos.json: Officer ID to name mapping and additional information if available.post_infos.json: Contains hierarchis of different components making up the postdept,role,juri,locaandstat, which map to Department, Rank, Jurisdiction, Location and Status.orders/*.order.json: Individual orders in json format.schema/*.schema.json: Schema information for all these json files can be found in thedata/schemadirectory. check out theREADME.mdfor an introduction.You can also Install the orgpedia_cabsec package, the package contains all the data created by this repository.python -m pip install orgpedia_cabsecOnce you install the package, all the data is available indata.zip. Use this command to get the path of thedata.zipinstalled on your computer.python -c "import pkg_resources;pkg_resources.resource_filename('orgpedia_cabsec', 'data.zip')"

<path/to/data.zip>Data StatsThese are high level statistics, please checkflowdirectory for more information.Number of Documents: 904Documents Processed: 817Number of Pages: 2,145Total Edits: 3,885Edits per Page:1.8112(3,885/2,145)Data ProcessingThis is a data package repository - it contains documents, configuration and code for processing the documents and creating data. In a sense it is different from code repositories that only contain code and not the artifacts the code generates.The data processing is broken down in series of Tasks, where each task processes the data created in the upstream task (links in theinputfolder) and generages new data stored in theoutputfolder. The directory layout of this repository follows the ideas mentioned in this video:Principled Data Processing by Patrick Ball. There are 3 main top-level directoriesimport,flowandexport. Asimplemakefileorchestrates the document flow across these folders, runmake helpto find out more about the commands.You can check out the template repositorytemplate.datapackagewhere each directory and sub-directory is explained. To understand how the data (/flow/buildTenure_/output) is generated from documents (/import/documents/) explore theflowdirectory.Deverloper NotesIf you want to make changes and regenerate data you have two choicesUse GitHub codespaces (WIP).Build locally, for this you will need at least 20 GB of space, as we store documents, intermediate data and final data locally. To minimize the space requirement it is recommended that you work only on the buildOrder/* and downstream tasks.Local DevelopmentPrerequisitesGit with Git LFSPython 3.7+PoetrymakeInstallationGit & Git LFSTo install Git, visit theGit websiteand follow the installation instructions for your operating system. For make sure Git-LFS stays enabled (default option). For othe platforms follow theseinstructionson Github.PythonTo install Python, visit thePython websiteand download the  version of Python 3.x for your operating system. Follow the installation instructions for your operating system.PoetryTo install Poetry, visit thePoetry websiteand follow installation instructions for your operating system:MakeOn Unix basedmakeshould come pre-installed, on Windows usewingetto installmake, follow instructionshere.SetupOrgpedia repository makes heavy use of soft-links, soft-links are stored in the GitHub repository. On non-windows platforms this is not a problem for Windows you need to do two things 1) enable soft-links and 2) tell git about it.Symlinks Setup On WindowsOn Windows 11, make sure you have enabled deverloper mode this will automatically enable soft-links on your machine. On windows 10 soft-links were added in Build 14972 and only works on Administrator cmd prompt. More info at thislink.Next you need to tell git it should create soft-links when it sees them in the respository, check the Stack Overflowanswerto know more about this. Execute the following command.git config --global core.symlinks trueTo setup the project, clone the repository using git (this is a large repository, will take several minutes):git clone https://github.com/orgpedia/cabsec.gitNavigate to the project directory:cd cabsecUse poetry to install software dependencies(one time only):make installImport models and other data-packages required for the document flow (one time only), these will be downloaded in theimportfolders and it takes a long time.make importGenerate DataAfter this you should have all the files needed to generate the data. Make whatever changes you need to make and then executemake flowThis will generate the data based on your changes. Currently, make does not track dependencies as a result the entire document flow is re-executed !!