Patent ID: 12217183

DETAILED DESCRIPTION OF THE INVENTION

The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which several embodiments of the invention are shown. This present disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art. Like numbers refer to like elements throughout, and base100reference numerals are used to indicate similar elements in alternative embodiments.

Referring initially toFIG.1, a tracking system10according to the present disclosure is now described. The tracking system10is for monitoring and tracking a plurality of containers (railcar container) within a railway yard18. As will be appreciated, as containers transit in and out of the railway yard18, it is critical to monitor the status and location of the containers to verify their proper routing. Moreover, there is a desire to monitor the movement of the containers to verify that safety protocols are being followed.

The tracking system10illustratively comprises a plurality of sensors11a-11cconfigured to generate sensor data. The plurality of sensors11a-11cmay comprise one or more of pressure sensors positioned on tracks, and motion sensors positioned on or adjacent to the tracks. The tracking system10illustratively includes a plurality of cameras12a-12cconfigured to generate image data, and a server13in communication with the plurality of sensors and the plurality of cameras. The plurality of cameras12a-12cmay comprise one or more different types of cameras, for example, pan tilt zoom (PTZ) cameras, fixed cameras, and night vision cameras (i.e. infrared cameras).

The server13illustratively includes a processor14and memory15cooperating therewith. The server13may comprise a device local to the railway yard18. In particular, the server13may be coupled to the plurality of sensors11a-11cand the plurality of cameras12a-12cover a local area network (LAN), for example, a wired LAN or a wireless local area network (WLAN). In these embodiments, the server13would also be coupled to the Internet to provide remote access.

In some embodiments, the server13may be provided within a cloud infrastructure, such as Amazon Web Services, Microsoft Azure, and the Google Cloud Platform. In these embodiments, the server13is coupled to the plurality of sensors11a-11cand the plurality of cameras12a-12cover the LAN and the Internet.

The processor14and memory15are configured to generate a database16associated with the plurality of containers based upon the sensor data and the image data. The database16may include a plurality of entries respectively associated with the plurality of containers. Each entry comprises a container type value, a container logo image, and a vehicle classification value. Of course, this list is merely exemplary, and the entry can include other data values, such as point of origin and destination. In essence, the server13is configured to perform data fusion operations on the sensor data and the image data to provide a snapshot of the containers in the railway yard18.

The processor14and memory15are configured to store the database16, and provide a user interface17to access the database. The user interface17may comprise a web interface accessible over the Internet.

Another aspect is directed to a method for operating a tracking system10for a plurality of containers within a railway yard18. The method includes operating a plurality of sensors11a-11cto generate sensor data, operating a plurality of cameras12a-12cto generate image data, and operating a server13in communication with the plurality of sensors and the plurality of cameras. The method comprises operating the server13to generate a database16associated with the plurality of containers based upon the sensor data and the image data. The database16includes a plurality of entries respectively associated with the plurality of containers. Each entry comprises a container type value, a container logo image, and a vehicle classification value. The method comprises operating the server13to store the database16, and provide a user interface17to access the database.

Referring now additionally toFIGS.2-4, another embodiment of the tracking system110is now described. In this embodiment of the tracking system110, those elements already discussed above with respect toFIG.1are incremented by 100 and most require no further discussion herein.

Referring toFIG.2, the container data flow process of the tracking system110is described. Here, the tracking system110illustratively includes a plurality of PTZ cameras112aand a plurality of fixed cameras112bconfigured to monitor the plurality of containers within the railway yard18. In particular, the plurality of fixed cameras112bmay be situated to monitor rails with the railway yard at strategic locations. The plurality of PTZ cameras112amay be used to more flexibly monitor the rest of the railway yard.

The tracking system110illustratively includes a PTZ module122coupled between the plurality of PTZ cameras112aand the server113. The PTZ module122is configured to control the plurality of PTZ cameras112aand route data flow therefrom. The tracking system110illustratively includes a choke module123coupled between the plurality of fixed cameras112band the server113and configured to manage the dataflow therebetween.

The tracking system110illustratively includes an analysis module125coupled to the database116, a messaging module126coupled to the analysis module, and a client121in communication with the messaging module. The analysis module125is configured to analyze data within the database116for slot events. The messaging module126is configured to generate messages based upon the output of the analysis module125and send appropriate messages to the client121.

In this embodiment, the server113comprises a C2I Intermodal Vision System (IVS) platform. In particular, the server113is configured to classify each container with a vehicle type, and a container type. The server113is also configured to process the image data from the pluralities of PTZ and fixed cameras112a,112bwith optical character recognition (OCR) processing to generate text strings, determine a color of each container, and generate image data associated with a logo carried by the container. For a respective container, using the text string, the logo image data, and the determined color, the server113is able to track and identify the respective container within the railway yard18.

Referring toFIG.3, the rail automation flow of the tracking system110is described. In this flow, the tracking system110illustratively includes a plurality of three dimensional (3D) sensors111(i.e. a sensor that provides 3D data regarding a sensed object), and an occupancy module127coupled between the plurality of 3D sensors and the server113. The occupancy module127is configured to manage the dataflow between the server113and the plurality of 3D sensors111and determine whether assigned tracks are currently occupied by containers.

The tracking system110illustratively includes a fixed module130coupled between the plurality of fixed cameras112band the server113and configured to manage the dataflow therebetween. The tracking system110illustratively includes a locomotive module131coupled to the database116and a user application module132also coupled to the database. The locomotive module131is configured to control movement of locomotives in the railway yard18. The user application module132is configured to process user requests for locomotive movement.

The tracking system110illustratively comprises a communications module133coupled to both the locomotive module131and the user application module132. Also, the tracking system110illustratively comprises a visual track occupancy system head end (VTOS-HE) module134coupled to the communications module133configured to receive messages from the communications module. For example, the messages may comprise a JavaScript Object Notation (JSON) format message. Of course, other message formats can be used. The server113is configured to determine railcar classification and track occupancy detection within the railway yard18.

Referring toFIG.4, the safety flow process of the tracking system110is described. Here, the tracking system110illustratively includes a safety module135coupled between the plurality of fixed cameras112band the server113. The server113is configured to monitor for illegal parking and speeding in the railway yard18, monitor for stop bar violations in the railway yard, monitor for safety gear violations in the railway yard, and monitor for abandoned/idled objects in the railway yard.

It should be appreciated that the server113may be situated in the cloud infrastructure for some embodiments. In these cloud embodiments, the controller modules (i.e. the PTZ module122, the choke module123, the occupancy module127, the fixed module130, the locomotive module131, the communications module133, and the safety module135) for data flows would be local to the railway yard18.

Referring now additionally toFIGS.5A-5D, flowcharts1000,1100,1200,1300illustrate rail service flow in an exemplary embodiment of the tracking system110. In particular, flowchart1000shows control of the plurality of cameras12a-12c, and how they are panned within the railway yard18. Also, the flowchart1000shows how images are sorted for OCR processing. Flowchart1100shows images are processed in sectional fashion, prioritizing the middle portion of the image. Flowcharts1200,1300show how the tracking system110parses and identifies vehicle classification values.

Referring now additionally toFIG.6, a flowchart1400illustrates the choke flow in an exemplary embodiment of the tracking system110. In particular, the tracking system110times streams from the plurality of cameras12a-12cbased upon the detection of events at the cameras.

Referring now additionally toFIG.7, a flowchart1500illustrates occupancy flow in an exemplary embodiment of the tracking system110. Again, the flowchart1500shows control of the plurality of cameras12a-12c, and how they are panned within the railway yard18. The images are fed into the occupancy detection algorithm, which comprises a machine learning component.

Referring now additionally toFIG.8, a diagram1600illustrates computational engines of an example embodiment of the tracking system110. In this embodiment, the computation engines illustratively comprise an interference model outputting downstream in a plurality of columns. The plurality of columns comprises a choke occupancy column, and a classification column.

Referring now toFIG.13, a control system200according to the present invention is now described. The control system200is for a railway yard201with a plurality of railroad tracks202a-202b. As will be appreciated, one or more of the features described above in the tracking system10,110may be incorporated into the control system200. The server207illustratively includes a processor214and memory215cooperating therewith.

The control system200illustratively includes a plurality of RCLs203a-203band sets of railcars204a-204c(e.g. a container, a railcar container, boxcar, a cargobeamer car, a coil car, a combine car, a flatcar, such as a container flatcar, a schnable car, a gondola car, a Presflo and Prestwin car, a bulk cement wagon car, a roll-block car, slate wagon car, stock car, a tank car, a tank wagon car or tanker, a milk car, a “Whale Belly” car, a transporter wagon car, and a well car),205a-205crespectively associated therewith on the plurality of railroad tracks202a-202b. As will be appreciated, each RCL203a-203bis capable of pushing or pulling a respective set of railcars204a-204c,205a-205con the plurality of railroad tracks202a-202b.

Each RCL203a-203billustratively comprises a geolocation device211(e.g. global positioning service (GPS) receiver) configured to generate a geolocation value for a respective RCL, a wireless transceiver212configured to enable remote control, and a controller213coupled to the wireless transceiver and the geolocation device. For example, the wireless transceiver212may comprise a cellular transceiver, or a WLAN transceiver (e.g. WiFi, WiMAX).

In some embodiments, each RCL203a-203bcomprises a plurality of onboard image sensors coupled to the controller213band generating a corresponding plurality of video streams. The controller213is configured to transmit the plurality of video streams to a remote control.

The control system200illustratively includes a plurality of railyard sensors206a-206cconfigured to generate railyard sensor data of the plurality of railroad tracks202a-202b. In some embodiments, the plurality of railyard sensors206a-206cmay comprise an image sensor configured to generate railyard image data. In other embodiments, the plurality of railyard sensors206a-206cmay comprise additionally or alternatively a proximity sensor (e.g. range finder sensor, pressure sensor) configured to detect a presence of plurality of RCLs203a-203b.

The control system200also includes a server207in communication with the plurality of RCLs203a-203band the plurality of railyard sensors206a-206c. The server207is configured to generate a database210associated with the sets of railcars204a-204c,205a-205cbased upon the railyard sensor data. The database210comprises, for each railcar204a-204c,205a-205c, a railcar type value, a railcar logo image, and a vehicle classification value.

The server207is configured to generate selectively control the plurality of RCLs203a-203bto position the sets of railcars204a-204c,205a-205cwithin the plurality of railroad tracks202a-202bbased upon the railyard sensor data. As will be appreciated, each RCL203a-203bcommunicates with the server207via the wireless transceiver212. The controller213is configured to transmit a plurality of operational values for the respective RCL203a-203bto the server207to provide snapshot of the current status. For example, the plurality of operational values comprises the geolocation value for the respective RCL203a-203b, but may include a speed value for the respective RCL, a bearing of the respective RCL, a condition of the respective RCL and associated sets of railcars204a-204c,205a-205c, and a weather condition of the railway yard201.

Also, the server207is configured to monitor the plurality of RCLs203a-203bfor railyard traffic violations. In particular, the railyard traffic violations may comprise illegal parking and speeding in the railway yard201, stop bar violations in the railway yard, safety gear violations in the railway yard, and abandoned/idled objects in the railway yard. Moreover, in some embodiments, the server207is configured to monitor the behavior of personnel within the railway yard201. For example, the server207is configured to monitor personnel for safety rule compliance.

The server207is configured to identify each railcar204a-204c,205a-205cbased upon the railyard image data. The server207is configured to perform OCR on the railyard image data. The server207is configured to perform machine learning on the railyard image data. In particular, the server207may be configured to execute a first machine learning model comprising a CNN trained to predict a location of text sequences in the railyard image data. The server207is configured to execute a second machine learning model comprising a RNN for scanning the text sequences and predicting a sequence of missing characters.

Yet another aspect is directed to a method for operating a control system200for a railway yard201with a plurality of railroad tracks202a-202b. The control system200includes a plurality of RCLs203a-203band sets of railcars204a-204c,205a-205crespectively associated therewith on the plurality of railroad tracks202a-202b, and a plurality of railyard sensors206a-206cconfigured to generate railyard sensor data of the plurality of railroad tracks. The method comprises operating a server207in communication with the plurality of RCLs203a-203band the plurality of railyard sensors206a-206cto generate a database210associated with the sets of railcars204a-204c,205a-205cbased upon the railyard sensor data, the database comprising, for each railcar, a railcar type value, a railcar logo image, and a vehicle classification value. The method includes operating the server207to selectively control the plurality of RCLs203a-203bto position the sets of railcars204a-204c,205a-205cwithin the plurality of railroad tracks202a-202bbased upon the railyard sensor data.

One of the most expensive challenges for train companies is maintaining and operating a railroad classification yard. The function of the railroad classification yard is to sort incoming trains in order to build outgoing trains. Within the yard there are rail cars and locomotives; the rail cars provide storage for the cargo to be shipped and the locomotive moves the train from point A to point B. Locomotives are also used to move rail cars within the railroad classification yard. Each of the controlled locomotives are commonly referred to as a rail control locomotive (RCL). Each RCL in the railroad classification yard typically contains a control module that controls the train's movement either in a forward direction or a backward direction.

The control module also controls the speed at which the train will move. The control system200uses VTOS-HE and provides railyard automation (RYA). The VTOS-HE is placed in an RCL and transmits instructions to the control module within the RCL regarding the direction and speed of the RCL. VTOS-HE can receive information from a remote location outside of the RCL and without the intervention of a human operator.

VTOS-HE is compatible with the VTOS system, which monitors the position, direction, and speed of individual rail cars, locomotives, and other debris or objects throughout the rail yard. VTOS-HE continuously utilizes the VTOS system to generate instructions to then be transmitted to its corresponding control module in the RCL. In addition, the VTOS-HE device can be mounted in an existing head end mounting bracket and meets all railroad standards for vibration and environmental regulations.

Furthermore, VTOS-HE will incorporate real time self-checking of all major components and subsystems. It will also have a positive indication of a failure and redundancies will be built into the system to protect against any accidents or catastrophic events.

The VTOS-HE system will incorporate a GPS so that the end user can pinpoint the location of the RCL within the classification yard. This control system200is intended to reduce injuries, fatalities, and costs associated with a railroad classification yard. In turn this control system200will provide a safer and less costly alternative to the typical approaches.

This control system200provides GPS coordinates and wirelessly connects automatically with a VTOS across a railroad classification yard. The railroad classification yard has a multitude of tracks upon which the individual locomotives move throughout the yard. This system will determine and report the speed and direction of a particular locomotive or remote control locomotive (RCL).

With this system it is designed so that locomotive in the classification yard of a railyard are operated remotely without the need for a human being present in the RCL. The VTOS system will use a pole with a light tree that is positioned in the classification yard. The light tree is used with the VTOS system. Additionally, a pole with a camera is positioned in the yard to provide images of the yard including any locomotives and will be able to document real time movement of individual locomotives in the yard. When the system is fully implemented multiple cameras may be used in the yard as well as multiple light trees.

With this system, control of the RCL is governed by commands from the VTOS-HE system and not by a human operator. Although the potential for human operation of the locomotive may exist, VTOS-HE can override the operator if the instructions are not followed.

VTOS-HE is intended to work in tandem with VTOS1when a locomotive is in the yard, approaching the yard, or departing from the yard. Once a locomotive is in either of the three locations VTOS-HE is automatically implemented and will immediately begin wirelessly receiving information from the VTOS system. The VTOS-HE unit is placed in the RCL to transmit information between the VTOS system and the other components of the VTOS-HE system. The information from VTOS will then be computed by VTOS-HE and then an instruction is quickly provided to the control module, provided by the RCL/RCO system. The control module actually controls the function of the locomotive.

Once the control module receives an instruction from VTOS-HE, it then carries out the instruction. For example, VTOS-HE may transmit an instruction to the control module or backhaul to slow down the locomotive 20 to 1 mph. The control module will then slow the locomotive to 1 mph.

In order to give the operator a visual reference a camera is incorporated into the VTOS system in order to provide visual images of the individual RCLs in the classification yard. This camera will operate in all lighting conditions and may include the capability to capture thermal imaging. However, an option is provided that allows an individual to manually override the instruction provided by VTOS-HE and submit his or her own instructions to the backhaul. This individual may be in the train or in a remote location. If an individual overrides VTOS-HE and inputs his or her own instructions or controls the locomotive manually, VTOS-HE will monitor and record how the train is controlled and the speed of the train.

Each locomotive in the yard containing a VTOS-HE system can transmit through VTOS the location of the locomotive it is installed into other VTOS head end units installed in other RCLs in the rail classification yard. The head end units then use that information along with the information provided by VTOS to generate and submit instructions to the corresponding control module in the RCL.

VTOS-HE will also collect mechanical information from the RCL/RCO system and transmit that information through VTOS. This is important for efficiency. If a locomotive has a mechanical malfunction, VTOS-HE will wirelessly transmit such information through VTOS. In turn the correct repairs can be performed quickly. Furthermore, the VTOS-HE can share this information through the VTOS network to other RCLs in the yard. Once informed, the VTOS-HE on other RCLs can instruct the control module accordingly so that a potential collision is avoided.

In addition to VTOS-HE receiving and transmitting information wirelessly, it will also incorporate real time self-checking of all major components and subsystems. It will also have positive indication of a failure and redundancies will be built into the system to protect against any accidents or catastrophic events.

The VTOS-HE system will be able to communicate with the RCL/RCO system via a serial data or TCP/IP communication protocol remotely and determine which track the RCL locomotive is occupying. In order to maximize the security of the system, the information that is transmitted in the VTOS-HE system may be encrypted. Additionally access to the system may be password protected. With this application multiple backhauls and multiple relays may be used to transmit information throughout the yard.

In the following, a discussion of exemplary features that may be included in the tracking system10,110and control system200now follows. The purpose of this document is to describe the utilities and capabilities of the IVS and its subservices.

Machine Learning Models

2.1. Stencil Recognition with OCR

The Stencil recognition engine provides OCR to the system. This engine uses two machine learning models in conjunction. The first model is a region-based convolutional neural networks (R-CNN) trained to predict the locations of text sequences in the image. The cropped text from each of these outputs is then given to a specialized RNN, which scans the text and predicts the sequence of characters it sees. This engine handles both horizontally- and vertically-oriented text, enabling it to predict on the text patterns commonly used on intermodal carriers.

2.2. Brand Recognition

The Brand Recognition Engine recognizes the shipper brand of a railcar or trailer. For example, JBHUNT, YRC, FedEx, UPS, etc.

2.3. Asset Type Recognition Engine

The purpose of the Asset Type Recognition Engine is to classify the type of asset. For example, “railcar/railcar container”, “container”, “trailer”, “Chassis”, “pickup truck”, “bobtail”, “hostler”, “pedestrian”, “worker” etc. In addition to the type of asset, the model also returns a bounding box surrounding the asset in the image.

2.4. Color Engine

The Color engine utilizes and reuses the R-CNN engine's localization to infer the color of the railcar or trailer's face. This engine can use both computer vision and machine learning to achieve this.

Using computer vision, the engine may return the most dominant colors in the cropped image as numeric coordinates in any given color space (e.g. RGB, HSV, YUV, LAB, etc.). Using machine learning, the engine may utilize a CNN trained to predict the dominant color of the image in human-readable form (e.g. “Red”, “Blue”, “White”, etc.).

2.5. PTZ Slot Classification Engine

The PTZ Slot Classification Engine is given an image snapshot from the PTZ camera. The image is then fed to an R-CNN model trained to detect the location and traits of the faces of railcars and trailers. The model also can determine the front and rear ends of the railcars and trailers.

This model aids the system in localizing the contents of the parking spot and classifies the carrier (identified by their logos, if any) and type of asset (See § 2.3). The classifications and especially the locations output by the R-CNN model are able to be used in other parts of the system to filter and/or validate the output predictions from other adjacent engines.

2.6. Unique Visual Characteristics Engine

Detect additional anomalies that may be used to further identify a particular asset, for example, rust or damage, graffiti, etc.

2.7. Weather Analysis Engine

The main purpose of the Weather Analysis Engine is to visually detect weather events that harm the systems operation, for example, heavy rain or fog.

2.8. Travel Direction Engine

The Travel Direction Engine determines the direction that an object is traveling with respect to the camera in an image or series of images.

2.9. Personnel Engine

The Personnel Engine determines if the image of a person contains specific characteristics required of every worker in the yard. This is accomplished by making use of an R-CNN model that has been trained to categorize the image of a person as to whether it contains the following features: Safety vest (existence); Safety vest (color); Hard hat (existence); Hard had (color); and Boots (existence).

PTZ Service

3.1. Service Overview

PTZ uses pan-tilt-zoom cameras as a way to survey the yard. Images from these cameras are fed through various engines to be reported to the database.

3.2. Service Operation

PTZ cameras are mounted throughout the yard. Pre-set locations are used to isolate spots to report codes. Each preset is meant to correlate to a specific location in the yard. The cameras cycle through their pre-set locations to continuously cycle through the yard's spots, taking a picture at each pre-set location. These pictures are sent to the PTZ Classification Module and OCR engine. After being processed by these engines, the OCR engine's code results are then weighted through the weighting system engines. The weighting engine system results can be filtered through various means. Codes not filtered are reported to the database, corresponding to the current pre-set location. Current filtration systems include: Classification R-CNN filter: codes read that are not the fronts of railcars are removed from the possible results from the spot in select locations. Priority code selection: codes read that are not central to the image are removed from the possible results. Neighbors Logic: remove codes that have been reported nearby to reduce repetition of codes.

3.3. PTZ Classification Module

The PTZ Classification module uses the PTZ Carrier Classification Engine, Color Engine, and unique visual characteristics engine to provide additional detail of the contents of a slot observed by PTZ, for example, “White JBHUNT with Rusted Top”, “Blue Trailer”, “Brown Stacked Chassis” etc.

3.4. Stencil Recognition with OCR

(See § 2.1).

3.5. Weighting

System for assigning confidence (weight) to OCR read codes of railcar stencils.Weight—Value representing the likelihood of a correct code match.OCR—Engine that takes images of characters and produces text representative of what the engine reads.Prefix—The first four values of a railcar code/stencil.
3.5.3. Inputs:OCR code—A code read by the OCR engine, from images provided by a camera. Current Inventory—A list of railcar codes in the yard obtained from the Database. Model Carrier Prediction—A prediction from a machine learning model that is designed to predict carriers. The model carrier prediction is then mapped to common carrier prefixes which are used in the stencil engine system.

Weighting will be composed of eight steps (Engines 18). Each step will attempt to provide a weight for an OCR generated code. If a step is successful, the weight, matched code and accompanying reasons list is returned, at which points the remaining steps will not occur. If it is not successful it continues onto the next step. In most cases, the OCR code is compared against the current inventory. Weighting/confidence values are variable to customer needs. Names are provided as a shorthand for the operations the engine performs. If no engine is successful in performing a match, a weight of zero is assigned, with the failure reasons of each engine attached.

3.5.5. Engine 1: Exact Match.

In this step, the OCR code is shortened to the first ten characters (or length of current inventory). It is then compared against a list of codes in current inventory. If the code matches exactly with any code in the inventory, this step is successful. The input is shortened, as the current inventory codes are stored as ten digit codes. On success: Returns a list of reasons that contains only an exact match confirmation. Returns the exact matching string. On failure: Reasons list adds an exact match failure description.

3.5.6. Engine 2: Unique Numbers Match.

In this step, the OCR code's numbers, except the check digit are compared against the numbers in the current inventory. If there is only one match then this step is successful. On success: Returns a list of reasons that contains an exact match failure and a unique numbers match success description. (See, e.g.,FIG.5C,95) Returns the unique matching string. On failure: List adds a unique numbers match failure description.

3.5.7. Engine 3: ML Prefix Augmentation

If the model carrier prediction is not empty, then the carrier prediction is correlated to a list of known carrier prefixes. If the shortened OCR code (the code without check digit), matches in nine out of ten of its values to a code in the current inventory, the carrier prefix list is used to substitute the OCR code's original prefix. Only prefixes within one character difference are attempted. If this creates an exact match in the inventory this step is successful. On success: returns a list of reasons that contains an exact match failure a unique numbers match failure and an ML Prefix Augmentation success description. (See, e.g.,FIG.5C,93) Returns the matching string after augmentation. On failure, list adds an ML Prefix Augmentation failure description.

3.5.8. Engine 4: ML Prefix Replacement

If the model carrier prediction is not empty, then the carrier prediction is correlated to a list of known carrier prefixes. The carrier prefix list is used to substitute the OCR code's original prefix. Only prefixes within one character difference are attempted. If this creates an exact match in the inventory this step is successful. On success: returns a list of reasons that contains an exact match failure, a unique numbers match failure, an ML Prefix Augmentation failure description, and an ML Prefix Replacement success. (See, e.g.,FIG.5C,91) Returns the matching string after Replacement. On failure: List adds an ML Prefix Replacement failure description.

Engine 5: Code Off by One.

If the shortened OCR code (the code without check digit), or the full ISO code matches in all but one of its characters to a code in the current inventory, this step is successful. On success, returns a list of reasons that contains failure reasons for above steps and a code off by one match success. (See, e.g.,FIG.5C,90) Returns the matching string. On failure, list adds a code off by one match failure description.

Engine 6: Numbers Only Match.

If the OCR code's numbers (excluding the check digit) are an exact match in the inventory, this step is successful. On success, returns a list of reasons that contain failure reasons for above steps and a numbers only match success. (See, e.g.,FIG.5C,60) Returns the full OCR code read without any modifications. On failure, list adds a numbers only match failure description.

3.5.11. Engine 7: ISO Check

If the code is compliant with ISO 6346, this step is successful. (See, e.g.,FIG.5C,45) On success, returns a list of reasons that contain failure reasons for above steps and an ISO check success. Returns the full OCR code read without any modifications. On failure, list adds an ISO Check failure description.

3.5.12. Engine 8: Prefix Match

If the prefix of the OCR code matches any prefix in the inventory, this step is successful. (See, e.g.,FIG.5C,40) On success, returns a list of reasons that contain failure reasons for above steps and an ISO check success. Returns the full OCR code read without any modifications. On failure, list returns all failure reasons encountered.

3.5.13. Stencil Engine Examples:

Each example case is followed by the engine that successfully handles the case. Code Read represents a stencil code read through OCR. All examples use this sample current inventory of the following codes: ABCD123456; QWER123457; WASD123457; and SWRU234567.

Example Case 1: Engine 1

Code Read: ABCD123456Match Condition: Exact string found in inventory.Code Returned: ABCD123456Explanation: The exact string exists in the inventory. Meeting criteria for Engine 1.

Example Case 2: Engine 2

Code Read: WASD123456Match Condition: Unique Numbers Match.Code Returned: ABCD123456Explanation: There is only one code in the current inventory with the numbers 123456. Meeting criteria for Engine 2.

Example Case 3: Engine 3

Code Read: KWER123457Machine Learning Prediction: railcar prefixes: QWER, DDCD.Match Condition: ML Prefix Augmentation.Code Returned: QWER123457Explanation: There is a code that is one value off of our prediction. After replacing with prefix QWER there is a match.QWER123457 is the only one in range to try replacement. Meeting criteria for Engine 3.

Example Case 4: Engine 4

Code Read: KPER123457Machine Learning Prediction: railcar prefixes: QWER, DDCD.Match Condition: ML Prefix Replacement.Code Returned: QWER123457Explanation: There is no code that is one value off of our prediction. After replacing with prefix QWER there is a match.QWER123457 and DDCD123457 are both attempted to be matched against inventory. Meeting criteria for Engine 4.

Example Case 5: Engine 5

Code Read: KWER123457Machine Learning Prediction: railcar prefixes: NONEMatch Condition: Code Off by One.Code Returned QWER123457Explanation: There is a code that is one value off of our prediction. There are no railcar prefixes to attempt to match with. The code is off by one from QWER123457. Meeting criteria for Engine 5.

Example Case 6: Engine 5

Code Read: KWER123457Machine Learning Prediction: railcar prefixes: ABCD, WARTMatch Condition: Code Off by One.Code Returned: QWER123457Explanation: There is a code that is one value off of our prediction. There are no matches after attempting prefix replacement. The code is off by one from QWER123457. Meeting criteria for Engine 5.

Example Case 7: Engine 5

Code Read: SWRU234568Machine Learning Prediction: railcar prefixes: NONEMatch Condition: Code Off by One.Code Returned: SWRU234567Explanation: There is a code that is one value off of our prediction. There are no railcar prefixes to attempt to match with. The code is off by one from SWRU234567. Meeting criteria for Engine 5.

Example Case 8: Engine 6

Code Read: TYUI123457Machine Learning Prediction: railcar prefixes: NONEMatch Condition: Numbers only match.Code Returned: TYUI123457Explanation: There is one match that has exactly the same numeral values. No railcar prefixes to attempt a match with. Sending read code. Meeting criteria for Engine 6.

Example Case 9: Engine 7

Code Read: CSQU3054383Machine Learning Prediction: railcar prefixes: NONEMatch Condition: ISO Valid.Code Returned: CSQU3054383Explanation: The code read is an ISO valid code. Meeting criteria for Engine 7.

Example Case 10: Engine 8

Code Read: WASD3054383Machine Learning Prediction: railcar prefixes: NONEMatch Condition: Prefix in inventory.Code Returned: WASD3054383Explanation: The code matches a prefix in current inventory.Meeting criteria for Engine 8.

Example Case 11: None

Code Read: WWW3054383Machine Learning Prediction: railcar prefixes: NONEMatch Condition: NONECode Returned: NONEExplanation: The code has not met any weighting conditions.Meets no engine criteria.
Choke Service
4.1. Service Overview

The Choke Service takes advantage of cameras set up at several “choke points” throughout the yard. Choke cameras are set up at entrances and exits to rows, and are sometimes also setup at various positions within a given row. The main idea of choke is to detect assets entering and leaving a particular section of the yard in order to gain further awareness of the movements and locations of assets. Choke cameras capture images of assets as they drive by and analyze the images to better understand the asset. Overall, the idea is for choke to understand, as deeply as possible, each asset that drives by.

4.2. Service Operation

4.2.1. Choke Camera

Wait for an object to enter the field of view. Capture a sequence of raw images while the object is in view. The sequence should start approximately at the time the object enters the view, and end at approximately the time the object leaves the view. Encode and save the sequence to filesystem storage. Send image sequence and relevant object detection information to the “choke consumer” for further analysis.

4.2.2. Choke Consumer

Wait for a sequence of images from the choke consumer. Send the image sequence to the Choke Classification Engine (See § 4.3) to detect relevant information regarding the assets in the image sequence. Stencil recognition with OCR: the sequence of images is sent to the OCR engine which returns all text strings found in the sequence of images as well as the bounding box of the detected text in image coordinates. (See § 2.5).

Background Asset Detection: many of the scenes viewed by the choke cameras contain stationary assets that should be ignored by choke. These are referred to “background assets”. Currently, background assets are detected based on the position of the OCR bounding box. However, additional methods including machine learning can be used as well.

Weighting: Strings return from OCR are sent to “weighting” the primary purpose of weighting is to match (potentially incomplete or slightly erroneous) OCR results to actual “current inventory” assets. (See § 3.5).

Direction analysis: currently a variety of methods are used to determine the direction the vehicle is traveling with respect to the camera. Object detection information from the camera is used to infer the direction of travel. If known, the previous location of the asset is also used to infer direction. For the travel direction engine. (See § 2.7). Publish all information gathered about the asset to our internal database and send a “Choke-Point” message to the customer via Kafka.

4.3. Choke Classification Engine

In this stage, a combination of models are used to form a more detailed description of the asset. The Asset Type Recognition Engine in combination with the Color Engine, Brand Recognition, and the Unique Visual Characteristics Engine create a detailed qualitative description of the asset. For example, “Gray FedEX railcar, no rust or damage, carried by a Black Bobtail”, “Red Trailer with No Logo, carried by a Yellow Hosler”, “White JBHUNT with a Rusted Top carried by a Blue Bobtail” etc.

4.4. Weighting

Engine 1: (See § 3.6.5).Engine 2: (See § 3.6.6).Engine 5: (See § 3.6.9).
Occupancy Service

Identifies the occupancy status of each spot in the intermodal yard. Occupancy R-CNN is robust in adverse weather and light conditions.

5.2. Service Operation

PTZ camera programmatically moves from one preset to another to cover different sections of the yard. PTZ camera snaps the Image for each preset. It also retrieves predefined box coordinates for that preset from the database which are sent to the occupancy module. The occupancy module is a trained R-CNN on specific images meant for determining if the spot is occupied. This module detects every railcar in the image and extracts coordinate positions of each railcar in a rectangular shape. Predicted boxes along with retrieved preset boxes are sent to mapping module.

Mapping module maps the predicted box to the known box to find the correct spot id of each detected railcar. Spots with mapped railcar box is labelled as occupied otherwise unoccupied.

5.3. Preparing Presets

Presets are taken in an optimized way to ensure minimum overlapping of spots exist between two consecutive presets. In case, multiple cameras from different poles can potentially cover the same region, pole with better coverage is given preference. Once preset is decided, every spot within preset was labeled with a spot id and a rectangular box covering the spot. Finally, this is repeated for each preset in different light and weather conditions to ensure robustness in varied weather conditions.

5.4. Object Detection Algorithm

R-CNN machine learning algorithm is trained to detect the railcars from images. Given an input image, it detects all the relevant object with their location in the image. The R-CNN algorithm is trained over thousands of training images with the goal of detecting railcars in input image with their locations.

5.5. Mapping Algorithm

Mapping algorithm consists of two sub-algorithms, drift-fix and rules to assign spots with occupancy decision (occupied/unoccupied). The drift-fix algorithm aligns the predicted boxes against the preset boxes by performing grid search. Search deemed successful when average Intersection over Union (IoU) of given image is greater than a predefined threshold value. Once drift is fixed, various rules are written to determine which spots are occupied. These rules are primarily based on the distance between predicted and ground truth box, and their IOUs. Spots that pass through occupancy logic are either labelled as occupied or unoccupied.

Choke-Occupancy Service

The choke occupancy service combines the functionality of the choke service and the occupancy service to monitor areas that the PTZ service cannot.

6.2. Service Operation

6.2.1. Tracking Camera Engine

The tracking camera engine monitors the region monitored by Choke Occupancy from a high level view. The purpose of the Tracking Camera Engine is to monitor vehicle traffic in the region to provide additional context to choke occupancy, such as the visual description (e.g. White JBHUNT carried by a Black Bobtail) as well as behavior and position information. An example output may be “Black UPS carried by a Black Bobtail drove straight through the region and did not park”, “White JBHUNT carried by a Blue Bobtail pulled over but did not park”, “Yellow Hostler picked up a Blue Amazon Trailer and left the Region”. Additionally, GPS data of the vehicle path is provided to further assist the Choke Occupancy Association Engine. All of the information from the Tracking Camera Engine supplements the choke and occupancy information to form a more complete picture of the events that take place in the region.

Operation: the tracking camera uses a static view to track and describe vehicles as they move through the region. Events are reported to the database to help choke occupancy narrow down potential matches between choke events and occupancy events. In some cases multiple cameras are required to monitor one Region, so the Tracking Camera Service is also prepared to synthesize data from multiple adjacent cameras, for example, a tracking data from one camera can be combined with tracking data from another camera to form a complete description of the objects path.

6.2.2. Choke Occupancy Association Engine

Each region is monitored by a choke region occupancy monitor that corresponds to a segment of the yard. A region monitor periodically associates choke events (passing assets) to occupancy changes using the following the steps. First, the database is queried for the choke events and occupancy changes for the current region. The database is queried for all tracking camera service events around the time of the choke and occupancy events. The next step is to disregard choke results that are younger than a determined minimum age, which are those that are unlikely to have already parked based on the time elapsed between the occupancy change and the choke event, as well as the distance from the choke camera to the spot.

The next step is to evaluate the compatibility of each choke result with each occupancy change using a combination of features including the color, brand, and additional characteristics of the railcar determined from choke and tracking. The time of the occupancy change and choke result are also considered, as well as the distance from the choke camera and the occupancy change. Tracking information is also considered, such as the path taken by the vehicle and whether or not the vehicle stopped, parked, or left the region. These considerations are used to create the best (lowest overall match cost) between the choked assets and known occupancy changes and reported to the database. Some choke occupancy regions do not have complete occupancy coverage. The following regions use a separate set of steps to associate choke events to the region. Query the database for the choke events for the current region. The next step is to disregard choke results that are younger than a determined minimum age. The remaining choke results are considered parked in the region and are assigned to every spot in the region.

Automatic Gate Service (AGS)

Identify characteristics of an incoming asset from the gate and associate said characteristics to that asset before adding it into inventory and allowing the asset to pass into the yard. Static cameras are set up at the entry gate to focus on areas of interest for incoming assets. Chassis Camera: lower portion of the asset where chassis codes are normally found. Truck Camera: front portion of the side of the asset where the truck is normally found. Railcar Camera: back portion of the side of the asset where the railcar/trailer is normally found. Top Camera: top of the asset to inspect the top of the railcar. Rear Camera: rear of the asset where the door to the railcar/trailer are normally located. Driver Camera: focused where a truck driver would normally be found. As an asset approaches the entry gate, it is forced to stop in view of the camera system.

Image from Chassis camera is classified by the following engines: Assert Type Recognition (See § 2.3); Stencil Recognition with OCR (See § 2.1); Image from the Truck Camera is classified by the following engines; Assert Type Recognition (See § 2.3); Stencil Recognition with OCR (See § 2.1).

Image from the railcar Camera is classified by the following engines: Assert Type Recognition (See § 2.3); Stencil Recognition with OCR (See § 2.1); Color (See § 2.4); Brand Recognition (See § 2.2); and Unique Visual Characteristics (See § 2.6).

Image from the Top Camera is classified by the following engines: Stencil Recognition with OCR (See § 2.1); and Unique Visual Characteristics (See § 2.6). Image from the Rear Camera is classified by the following engines: Stencil Recognition with OCR (See § 2.1); Brand Recognition (See § 2.2); and Unique Visual Characteristics (See § 2.6).

Image from the Driver Camera is simply stored in image data store. These characteristics are associated to the incoming asset and the asset is added to the current yard inventory. The AGS system determines where the asset should be parked and informs the driver of where to park. The asset is then allowed to pass into the yard in order to park where instructed.

Security and Safety Service

The security and safety service identifies instances of employees and other individuals performing actions that are deemed harmful to the intermodal operation.

8.2.1. Speeding

Position the camera so that the field of view contains a large enough portion of the area of interest. Determine the two speeding alarm zones. These alarm zones are positioned at opposite sides of the field of view. Stream video frames from the camera while determining and tracking objects in the scene using object detection and tracking algorithms. Monitor object detections and alarm zones to determine speeding violations.

A speeding violation is determined by the following criteria: the object detection has entered both alarm zones; the difference between the before alarm zone time and the after alarm zone time for the object detection is within the specified threshold; and the time threshold is determined by the position of the camera and the speed limit in the monitored area.

8.2.2. Stop Bar

Position the camera so that the stop bar line is located at the opposite end of where the monitored vehicles enter the field of view. Determine the before and after stop bar alarm zones. The before alarm zone is positioned at the vehicle entrance point in the field of view. The after stop bar alarm zone is positioned between the stop bar and the end of the field of view. Stream video frames from the camera while determining and tracking objects in the scene using object detection and tracking algorithms. Monitor object detections and alarm zones to determine stop bar violations. A stop bar violation is determined by the following criteria: the object detection has entered both alarm zones; the object detection entered the before alarm zone prior to the after alarm zone ensuring correct direction; the difference between the before alarm zone time and the after alarm zone time for the object detection is within the specified threshold; and the time threshold is determined by the position of the camera and the speed at which the detection should pass between both alarms.

8.2.3. Personnel

This section describes the engine that detects and reports features of personnel. Certain safety regulations are set for different areas of the intermodal yard. Drivers, contractors, and employees of the yard have different requirements to adhere to in order to make sure the workplace is kept as safe as possible. The safety-personnel engine's purpose is to detect the absence or existence of particular clothing and other equipment and make a determination if safety rules are being violated. Depending on the area and type of employee, a determination is made as to whether or not this may break a safety regulation. If a safety regulation was likely broken, the image and type of violation would be saved and reported.

Database

The design of the database is crucial for all the modules due to the fact that it works as a repository for the configuration, information being generated by the applications and logs as well. A proper design requires for the database for the server to get response times less than a second for any of the request of the multiple modules.

The following is the schema diagrams for the different modules in the database: Configuration schema diagram1700inFIGS.9A and9B; Module's information schema diagram1800inFIGS.10A,10B, and10C; logs diagram1900inFIG.11. It should be appreciated that these schemas may change over time.

Infrastructure: in order to achieve the requirements for the database service, a layout like the following is put in place for any module: The client2000represents the system in which our application run: (FIG.12). In this case, this are multiple virtual environments. The app module2001represents each individual module from where the information is being produced and consumed. The PgBouncer module2002is a proxy that routes the connections of the app module2001to and from the database server2003. The proxy manages the multiple connections in a faster and smarter way adding an extra layer of security due to the fact that the database is hidden from the app side. The database represents the virtual environment where the database service is running handling the connections being produced by the proxy PgBouncer module2002.

Many modifications and other embodiments of the present disclosure will come to the mind of one skilled in the art having the benefit of the teachings presented in the foregoing descriptions and the associated drawings. Therefore, it is understood that the present disclosure is not to be limited to the specific embodiments disclosed, and that modifications and embodiments are intended to be included within the scope of the appended claims.