Archived data crawling

Described is a content mining system comprising a crawler configured to retrieve a plurality of files from a data storage system. The content mining system further comprises a plurality of converters configured to extract data from the plurality of files retrieved by the crawler from the data storage system, where each of the plurality of converters is configured to process a respective type of data. The content mining system further comprises a plurality of queues interposed between the crawler and the plurality of converters, where each queue is associated with a single converter.

BACKGROUND

The present disclosure relates to data mining, and, more specifically, to processing crawled data using a plurality of queues respectively associated with a plurality of converters in a content mining system.

Data crawling, data scraping, web crawling, and web scraping all refer to aggregating large amounts of data. Data can be aggregated from an intranet, the Internet, production data storage systems, backup data storage systems, and/or archival data storage systems. The aggregated data can be used for analytics, machine learning, model building, and/or Artificial Intelligence training. More generally, aggregating large amounts of data from disparate sources can be useful for generating insights and/or predictions that would not otherwise be possible.

SUMMARY

Aspects of the present disclosure are directed toward a content mining system comprising a crawler configured to retrieve a plurality of files from a data storage system. The content mining system further comprises a plurality of converters configured to extract data from the plurality of files retrieved by the crawler from the data storage system, where each of the plurality of converters is configured to process a respective type of data. The content mining system further comprises a plurality of queues interposed between the crawler and the plurality of converters, where each queue is associated with a single converter.

Further aspects of the present disclosure are directed toward a computer-implemented method of aggregating, by a crawler, a plurality of files with a plurality of file types from a data storage system. The method further comprises providing the plurality of files to a plurality of queues interposed between the crawler and a plurality of converters, where files with a respective file type are sent to respective queues corresponding to a converter configured to process the files with the respective file type. The method further comprises extracting data from the plurality of files by the plurality of converters, where the plurality of converters incrementally process the plurality of files using the plurality of queues.

Additional aspects of the present disclosure are directed to systems and computer program products configured to perform the method described above. The present summary is not intended to illustrate each aspect of, every implementation of, and/or every embodiment of the present disclosure.

DETAILED DESCRIPTION

Aspects of the present disclosure are directed toward to data mining, and, more specifically, to processing crawled data using a plurality of queues respectively associated with a plurality of converters in a content mining system. While not limited to such applications, embodiments of the present disclosure may be better understood in light of the aforementioned context.

Software products such as Watson Explorer and Watson Discovery Advisor® (products of International Business Machines Corporation) can be used to analyze documents and/or data for content mining. Content mining products can evaluate various kinds of documents and/or files by extracting data from the various kinds of documents and/or files using different types of converters. For example, various converters can extract text data and metadata using particular techniques for particular file types.

As one example, a Comma Separated Values (CSV) converter can extract data from CSV files by dividing data into lines. As another example, a JavaScript® Object Notation (JSON) converter can extract data from JSON files by dividing arrays in the JSON file. As yet another example, a HyperText Markup Language (HTML) converter can extract data from HTML files by analyzing an HTML tag to extract text data and a title from the HTML files. As yet another example, a Smart Document Understanding (SDU) converter can extract data from document files such as Portable Document Format (PDF), Microsoft Word files, Microsoft Excel files, Microsoft PowerPoint files, and the like.

However, the crawler collecting data and the converters extracting data each process data at different rates. Specifically, the converters use more CPU and memory capacity compared to the crawlers, and this can result in a slower processing rate of the converters relative to the crawlers. As a result, a queue can be placed between the crawler and the converters to store an accumulation of crawled data that is ready for data extraction by the converters.

As previously discussed, different converters can be used for different file types. Disadvantageously, if there is a consecutive series of crawled data with a same file type, then converters corresponding to other file types can be left unused while the queue sequentially works through the consecutive series of crawled data with the same file type. In other words, a single queue feeding a plurality of converters can result in inefficient processing of crawled data insofar as some converters may experience significant downtime while waiting for the single queue to work through a consecutive series of a certain type of file.

In one workaround to the above disadvantage, the converters can be dynamically changed to work through the consecutive series of crawled data with a same file type. However, different converters have different CPU and/or memory requirements, thereby making it difficult to estimate CPU and/or memory required to sufficiently resource the dynamically modified converters. In other words, a queue for a first file type that is dynamically modified to extract data from a second file type can be under resourced (and thereby inefficient) at effectively extracting data from the second file type.

In another workaround to the above disadvantage, the crawler can be suspended when the queue exceeds a predetermined threshold and restarted when the queue falls below the predetermined threshold. However, in the case of a crawler collecting data from a tape storage system, while data is read from the tape, physical movement of the head cannot be stopped even when the reading can be temporarily suspended. In this case, repositioning of the head will occur when resuming reading, and thus it takes longer to restart reading. Thus, suspending the crawler is inefficient due to both the downtime of the suspended crawler and the inefficiencies introduced by moving the head during restarts in tape storage systems. It is possible to throttle the crawler to collect data at a slower speed, where the slower speed matches a capacity of a single queue to work through a consecutive series of similar file types (whereby throttling the crawler reduces or removes the need to suspend it and the associated start/stop inefficiencies). However, throttling the crawler is obviously inefficient.

Aspects of the present disclosure are directed to overcoming the aforementioned disadvantages and deficient workarounds. In some embodiments, aspects of the present disclosure read and crawl data in one direction on a tape storage system to avoid rewinding tap. Furthermore, aspects of the present disclosure can provide a plurality of queues for different file types to avoid an overaccumulation of a sequence of a same file type in a single queue feeding to plurality of converters.

Referring now to the figures,FIG.1illustrates a block diagram of an example computational environment100for content mining using a plurality of queues108interposed between a crawler104and a plurality of converters110, in accordance with some embodiments of the present disclosure. The computational environment100for content mining can include a content mining system124. The content mining system124can use crawler104to generate crawled data106that is retrieved from a data source server102. The crawler104can be a software module configured to access a data storage system and crawl documents and/or files from the accessed data storage system. In some embodiments, the data source server102can include a content management system, a file server, a tape storage system, a disk storage system, or another data storage system that stores files and/or documents.

The crawler104can pass the crawled data106to a plurality of queues108(e.g., queue1108-1, queue2108-2, and/or queue N108-N where N is any integer representing any number of queues). The queues108can be respectively associated with converters110(e.g., converter1110-1, converter2110-2, and converter M110-M, where M is any integer representing any number of converters). In some embodiments, there is an equal number of queues108and converters110(e.g., N=M) such that each queue108is associated with one and only one converter110. The converters110can be software modules configured to extract text data and/or metadata (e.g., file names) from the crawled data106.

In some embodiments, respective converters110are dedicated to extracting data from certain file types (e.g., converter1110-1is dedicated to CSV files, converter2110-2is dedicated to JSON files, and the like). In such embodiments, respective queues108can be customized to a particular size based on processing speed characteristics for the corresponding converter110(discussed in more detail hereafter with respect toFIG.2). Whatever the size of the queues108, the queues108can incrementally pass accumulated files to a corresponding converter110for processing.

The converters110can pass extracted data to enrichment112. Enrichment112can be a software module configured to analyze extracted data and add information to the extracted data (e.g., parts of speech (POS) tagging, sentiment, etc.). The enriched data generated by enrichment112can be passed to indexer114. Indexer114can be a software module configured to create an index116for searching content of the enriched data received from enrichment112. The index116can be accessed by a user120via a search server122. The search server122can match search parameters provided by the user120to relevant data118in the index116. The search server122can then return the relevant data118to the user120.

Advantageously, the content mining system124realizes improved efficiencies relative to other content mining systems. For example, the index116generated from the crawled data106can be built faster insofar as a plurality of queues108are used, where the plurality of queues108are respectively associated with converters110configured to extract data from predetermined file types. By using the plurality of queues108, aspects of the present disclosure overcome queue backlogs realized by a single queue processing a sequence of files with a same file type. Furthermore, by using the plurality of queues108, aspects of the present disclosure do not need to throttle or suspend the crawler104. This is advantageous when the crawler104includes, at least in part, a head of a tape drive system that exhibits increased inefficiency with increasing amounts of interrupted (e.g., start/stop) reading.

FIG.2illustrates a block diagram of a subsystem200having queues interposed between a crawler and a plurality of converters in a content mining system, in accordance with some embodiments of the present disclosure. In some embodiments, crawler202is consistent with crawler104ofFIG.1. In some embodiments, CSV queue204, JSON queue208, HTML queue212, and SDU queue216are consistent with queues108ofFIG.1. In some embodiments, CSV converter206, JSON converter210, HTML converter214, and SDU converter218are consistent with converters110ofFIG.1.

Subsystem200can be a subsystem of a content mining system such as content mining system124ofFIG.1. In some embodiments, subsystem200can be included in any product configured to analyze, process, read, evaluate, organize, or otherwise manipulate large amounts of data and using various converters.

The crawler202can send crawled data to each of the queues such that a file type of the crawled data matches the file type the corresponding converter is configured to process. For example, crawler202can send CSV files to CSV queue204, and the CSV queue204can feed the accumulated CSV files to the CSV converter206. Similarly, the crawler202can send JSON files to JSON queue208, and JSON queue208can feed the accumulated JSON files to the JSON converter210. Further, the crawler202can send HTML files to HTML queue212, and HTML queue212can feed the accumulated HTML files to the HTML converter214. Further, the crawler202can send other predetermined files (e.g., PDF files, Microsoft Word files, Microsoft Excel files, Microsoft PowerPoint files, etc.) to SDU queue216, and SDU queue216can feed the accumulated files to the SDU converter218. As will be appreciated by one skilled in the art, the aforementioned types of queues, converters, and files are for example purposes only, and should not be construed as limiting. In other embodiments, more, fewer, and/or different queues, converters, and/or files can be used while remaining within the spirit and scope of the present disclosure.

The sizes of CSV queue204, JSON queue208, HTML queue212, and SDU queue216can be customized based on the processing time of the corresponding converter. For example, hypothetical processing times of various converters are shown in Table 1:

The sizes of CSV queue204, JSON queue208, HTML queue212, and SDU queue216can be customized based on the processing times shown in Table 1 such that converters with relatively shorter (e.g., faster) processing times per file can have a relatively larger number of files queued in their corresponding queue, and, conversely, converters with relatively longer (e.g., slower) processing times per file can have a relatively smaller number of files queued in their corresponding queue. As an example, Table 2 illustrates example queue sizes for the various converters based on an algorithm such as dividing one hundred by the average processing time:

As can be seen from Table 1 and Table 2, queue size can be inversely related to the processing speed of the corresponding converter. Said another way, a first queue associated with a first converter having a larger (e.g., slower) processing speed will be configured to store fewer files than a second queue associated with a second converter having a faster (e.g., smaller) processing speed.

FIG.3illustrates a flowchart of an example programmatic method300for processing crawled data using a plurality of queues respectively associated with a plurality of converters, in accordance with some embodiments of the present disclosure. The method300can be implemented by, for example, a content mining system124ofFIG.1, a subsystem200ofFIG.2, a computer, a server, a processor, or another combination of hardware and/or software.

Operation302includes crawling data. Operation302can be performed by, for example, a crawler such as crawler104ofFIG.1or crawler202ofFIG.2. In some embodiments, operation302includes preparing, by the crawler, a list (e.g., L1) by arranging all files on a tape drive in order based on the starting offset as shown in Table 3:

As shown in Table 3, files can be arranged in column “#”, a path for each file can be specified, and a starting offset can be specified. Further, whether files are already crawled is recorded in the “ingested” column (where a flag such as “N” can refer to not crawled and/or a flag such as “Y” can refer to already crawled). As shown in Table 3, there can be a total of NL files to be crawled.

In some embodiments, the data shown in Table 3 is aggregated from data stored in one or more tape drives of a tape drive storage system. In some embodiments, the one or more tape drives can be formatted in a Linear Tape File System (LTFS) format. Using the LTFS format, tape drives can be managed and accessed using, for example, Spectrum Archive Single Drive Edition (SDE) (a product of International Business Machines Corporation). In particular, a file name (e.g., “#” in Table 3) and a position on a tape (e.g., the starting offset shown in Table 3) at which respective files are stored can be obtained using an Application Programming Interface (API) tape management command such as POSIX.

In Linear Tape-Open (LTO) tapes supported by LTFS, a width direction can be divided into wraps and each wrap stores data from the beginning to the end and from the end to the beginning in a longitudinal direction. In other words, LTO tapes supported by LTFS can write data in the direction from the beginning to the end, and, upon reaching the end, switch the wrap to write data in the direction from the end to the beginning. As a result, aspects of the present disclosure are useful for increasing efficiency in crawling and converting large amounts of tape drive data insofar as (i) a plurality of queues are used (to reduce the need to suspend/restart the tape drive head), and (ii) the files are read by sequentially ordered starting offset (for efficient movement of the tape drive head).

Operation304includes setting i=0 (e.g., beginning with an initial file illustrated in Table 3). Operation306includes determining if the selected file has already been crawled (e.g., by querying the “ingested” column of Table 3). If not (306: NO), then the method300proceeds to operation308where a file type can be retrieved from the file extension of the selected file.

Operation310can include determining whether or not the queue corresponding to the retrieved file type is full (e.g., CSV queue204for a CSV file, JSON queue208for a JSON file, HTML queue212for a HTML file, SDU queue216for a predefined document file, or another queue for a different type of file). If not (310: NO), then the method300proceeds to operation312and reads the file and adds the file to the appropriate queue. Operation314can then include marking the file as crawled (e.g., by updating a value in the “ingested” column of Table 3).

After operation314, the method can proceed to operation316. Furthermore, if the queue is determined to be full in operation310(e.g.,310: YES) or if the selected file is determined to have previously been crawled in operation306(e.g.,306: YES), then the method300proceeds directly to operation316. Operation316includes iterating to a next file. In some embodiments, operation316queries Table 3 to identify the next file, where the next file can be a file with a next starting offset to a current offset in a tape drive system.

Operation318includes determining if the next file is greater than NL. In other words, operation318determines if the end of the list of files shown in Table 3 is reached. If not (318: NO), then the method returns to operation306with the next iterated file as the selected file. If so (318: YES), then the method300proceeds to operation320and determines if files in list L1 are all crawled (e.g., by determining whether or not every file is associated with a crawled flag in the “ingested” column of Table 3). If so (320: YES), then the method300ends. If not (320: NO), then the method300returns to operation304and iterates through the files again (starting with an initial file) until all files are crawled.

FIG.4illustrates a flowchart of an example general method400for processing crawled data using a plurality of queues respectively associated with a plurality of converters, in accordance with some embodiments of the present disclosure. The method400can be implemented by, for example, a content mining system124ofFIG.1, a subsystem200ofFIG.2, a computer, a server, a processor, or another combination of hardware and/or software.

Operation402includes aggregating a plurality of files with a plurality of file types from a data storage system. In some embodiments, operation402is performed by a crawler of a content mining system. In some embodiments, the data storage system comprises a tape drive storage system. In such embodiments, the crawler can iterate through files in the tape drive storage system according to respective offsets of the files (e.g., sequential starting offset values) in order to reduce head movement of the tape drive storage system.

Operation404includes providing the plurality of files to a plurality of queues interposed between the crawler and a plurality of converters. In some embodiments, respective queues have a customized size (e.g., number of files of a specific file type they can hold) based on a processing speed of the corresponding converter for the specific file type. In other words, in some embodiments, queue sizes are inversely proportional to the processing speed of the corresponding converters (e.g., slower (larger) processing times result in a queue configured to hold fewer files while faster (shorter) processing times result in a queue configured to hold more files).

Operation406includes extracting data from the plurality of files by the plurality of converters. Advantageously, each of the plurality of queues can respectively provide files to the plurality of converters to reduce downtime (thereby increasing throughput and efficiency) of the plurality of converters and the crawler.

FIG.5illustrates a block diagram of an example computer500in accordance with some embodiments of the present disclosure. In various embodiments, computer500can perform any or all portions of the methods described inFIGS.3-4and/or implement the functionality discussed inFIGS.1-2. In some embodiments, computer500receives instructions related to the aforementioned methods and functionalities by downloading processor-executable instructions from a remote data processing system via network550. In other embodiments, computer500provides instructions for the aforementioned methods and/or functionalities to a client machine such that the client machine executes the method, or a portion of the method, based on the instructions provided by computer500. In some embodiments, the computer500is incorporated into (or functionality similar to computer500is virtually provisioned to) one or more entities illustrated inFIGS.1-2and/or other aspects of the present disclosure.

Each CPU505retrieves and executes programming instructions stored in memory525or storage530. Interconnect520is used to move data, such as programming instructions, between the CPUs505, I/O device interface510, storage530, network interface515, and memory525. Interconnect520can be implemented using one or more buses. CPUs505can be a single CPU, multiple CPUs, or a single CPU having multiple processing cores in various embodiments. In some embodiments, CPU505can be a digital signal processor (DSP). In some embodiments, CPU505includes one or more 3D integrated circuits (3DICs) (e.g., 3D wafer-level packaging (3DWLP), 3D interposer based integration, 3D stacked ICs (3D-SICs), monolithic 3D ICs, 3D heterogeneous integration, 3D system in package (3DSiP), and/or package on package (PoP) CPU configurations). Memory525is generally included to be representative of a random-access memory (e.g., static random-access memory (SRAM), dynamic random-access memory (DRAM), or Flash). Storage530is generally included to be representative of a non-volatile memory, such as a hard disk drive, solid state device (SSD), removable memory cards, optical storage, or flash memory devices. In an alternative embodiment, storage530can be replaced by storage area-network (SAN) devices, the cloud, or other devices connected to computer500via I/O device interface510or network550via network interface515.

In some embodiments, memory525stores instructions560. However, in various embodiments, instructions560are stored partially in memory525and partially in storage530, or they are stored entirely in memory525or entirely in storage530, or they are accessed over network550via network interface515.

Instructions560can be computer-readable and computer-executable instructions for performing any portion of, or all of, the methods ofFIGS.3-4and/or implement the functionality discussed inFIGS.1-2. Although instructions560are shown in memory525, instructions560can include program instructions collectively stored across numerous computer-readable storage media and executable by one or more CPUs505.

In various embodiments, I/O devices512include an interface capable of presenting information and receiving input. For example, I/O devices512can present information to a user interacting with computer500and receive input from the user.

Characteristics are as follows:

Service Models are as follows:

Deployment Models are as follows:

Embodiments of the present invention can also be delivered as part of a service engagement with a client corporation, nonprofit organization, government entity, internal organizational structure, or the like. These embodiments can include configuring a computer system to perform, and deploying software, hardware, and web services that implement, some or all of the methods described herein. These embodiments can also include analyzing the client's operations, creating recommendations responsive to the analysis, building systems that implement subsets of the recommendations, integrating the systems into existing processes and infrastructure, metering use of the systems, allocating expenses to users of the systems, and billing, invoicing (e.g., generating an invoice), or otherwise receiving payment for use of the systems.

Different instances of the word “embodiment” as used within this specification do not necessarily refer to the same embodiment, but they can. Any data and data structures illustrated or described herein are examples only, and in other embodiments, different amounts of data, types of data, fields, numbers and types of fields, field names, numbers and types of rows, records, entries, or organizations of data can be used. In addition, any data can be combined with logic, so that a separate data structure may not be necessary. The previous detailed description is, therefore, not to be taken in a limiting sense.

Any advantages discussed in the present disclosure are example advantages, and embodiments of the present disclosure can exist that realize all, some, or none of any of the discussed advantages while remaining within the spirit and scope of the present disclosure.

A non-limiting list of examples are provided hereinafter to demonstrate some aspects of the present disclosure. Example 1 is a content mining system comprising: a crawler configured to retrieve a plurality of files from a data storage system; a plurality of converters configured to extract data from the plurality of files retrieved by the crawler from the data storage system, wherein each of the plurality of converters is configured to process a respective type of data; and a plurality of queues interposed between the crawler and the plurality of converters, wherein each queue is associated with a single converter.

Example 2 includes the content mining system of example 1, including or excluding optional features. In this example, each queue of the plurality of queues is respectively sized based on a processing speed of a corresponding converter. Optionally, the queue sizes are inversely related to converter processing speeds. Optionally, a first queue corresponding to a first converter with a first processing speed for a first file type is sized to store a first number of files of the first file type, wherein a second queue corresponding to a second converter with a second processing speed for a second file type is sized to store a second number of files of the second file type, wherein the first processing speed is greater than the second processing speed, and wherein the first number of files is less than the second number of files.

Example 3 includes the content mining system of any of examples 1 to 2, including or excluding optional features. In this example, the data storage system is a tape drive storage system. Optionally, the crawler is further configured to iterate through the plurality of files on the tape drive storage system based on sequential starting offsets of the files.

Example 4 includes the content mining system of any of examples 1 to 3, including or excluding optional features. In this example, the plurality of converters include at least a Comma Separated Value (CSV) converter, a JavaScript® Object Notation (JSON) converter, and a HyperText Markup Language (HTML) converter.

Example 5 is a computer-implemented method comprising aggregating, by a crawler, a plurality of files with a plurality of file types from a data storage system; providing the plurality of files to a plurality of queues interposed between the crawler and a plurality of converters, wherein files with a respective file type are sent to respective queues corresponding to a converter configured to process the files with the respective file type; and extracting data from the plurality of files by the plurality of converters, wherein the plurality of converters incrementally process the plurality of files using the plurality of queues.

Example 6 includes the method of example 5, including or excluding optional features. In this example, each queue of the plurality of queues is respectively sized based on a processing speed of a corresponding converter. Optionally, queue sizes are inversely related to converter processing speeds. Optionally, a first queue corresponding to a first converter with a first processing speed for a first file type is sized to store a first number of files of the first file type, wherein a second queue corresponding to a second converter with a second processing speed for a second file type is sized to store a second number of files of the second file type, wherein the first processing speed is greater than the second processing speed, and wherein the first number of files is less than the second number of files.

Example 7 includes the method of any of examples 5 to 6, including or excluding optional features. In this example, the data storage system is a tape drive storage system. Optionally, the crawler is further configured to iterate through the plurality of files on the tape drive storage system based on sequential starting offsets of the plurality of files.

Example 8 includes the method of any of examples 5 to 7, including or excluding optional features. In this example, the plurality of converters include at least a Comma Separated Value (CSV) converter, a JavaScript® Object Notation (JSON) converter, and a HyperText Markup Language (HTML) converter.

Example 9 includes the method of any of examples 5 to 8, including or excluding optional features. In this example, the method is performed by one or more computers according to software that is downloaded to the one or more computers from a remote data processing system. Optionally, the method further comprises: metering a usage of the software; and generating an invoice based on metering the usage.

Example 10 is a system. The system comprises one or more computer readable storage media storing program instructions; and one or more processors which, in response to executing the program instructions, are configured to perform a method according to any one of examples 5 to 9.

Example 11 is a computer program product. The computer program product comprises one or more computer readable storage media, and program instructions collectively stored on the one or more computer readable storage media, the program instructions comprising instructions configured to cause one or more processors to perform a method according to any one of examples 5 to 9.