A computer-implemented method of processing digital content via a content processing management system, the method comprising: launching, at a plurality of computing devices, a plurality of processing nodes, each processing node including at least one tag that determines which job type can be performed by the processing node; fetching, by the content processing management system, a plurality of content sources, each content source being associated with one or more tags for that content source; initiating, by the content processing management system, one or more content processing scans of one or more content sources by creating one or more start jobs in a queue stored in a job database, the jobs being tagged to the content source being scanned; grouping, at the content processing management system, the jobs into batches based on the content source associated with the job such that all jobs in a single batch have the same tag; receiving, from a first processing node and at the content processing management system, a request for a batch of one or more jobs with at least a first tag matching the tags of the first processing node; providing, from the content processing management system, a first batch of jobs to the first processing node, each job in the first batch of jobs including a tag matching the first tag; processing, at the first processing node, each job via a set of pipelines, wherein each pipeline of the set of pipelines is configured to fetch and process content according to a defined list of tasks; adding, at the first processing node and via a database provider, new sub-jobs to be processed to the job database for each job being processed; grouping, at the content processing management system, batches of newly added sub-jobs based on their tags; and providing, at the content processing management system, batches of newly added sub-jobs to processing nodes when requested, such that the jobs provided match the tags of the corresponding requesting processing node, until there are no more jobs remaining to be processed; wherein each tag is an identifier that determines where a job should be processed. launching, at a plurality of computing devices, a plurality of processing nodes, each processing node including at least one tag that determines which job type can be performed by the processing node; fetching, by the content processing management system, a plurality of content sources, each content source being associated with one or more tags for that content source; initiating, by the content processing management system, one or more content processing scans of one or more content sources by creating one or more start jobs in a queue stored in a job database, the jobs being tagged to the content source being scanned; grouping, at the content processing management system, the jobs into batches based on the content source associated with the job such that all jobs in a single batch have the same tag; receiving, from a first processing node and at the content processing management system, a request for a batch of one or more jobs with at least a first tag matching the tags of the first processing node; providing, from the content processing management system, a first batch of jobs to the first processing node, each job in the first batch of jobs including a tag matching the first tag; processing, at the first processing node, each job via a set of pipelines, wherein each pipeline of the set of pipelines is configured to fetch and process content according to a defined list of tasks; adding, at the first processing node and via a database provider, new sub-jobs to be processed to the job database for each job being processed; grouping, at the content processing management system, batches of newly added sub-jobs based on their tags; and providing, at the content processing management system, batches of newly added sub-jobs to processing nodes when requested, such that the jobs provided match the tags of the corresponding requesting processing node, until there are no more jobs remaining to be processed; wherein each tag is an identifier that determines where a job should be processed.