Patent Description:
The present invention relates in general to the field of railroad maintenance. More particularly, the present invention relates to a computer vision-based railway component detector system that detects various components associated with a railway. For instance, the present invention may be used to detect specific components associated with a railway, for instance spike holes, after which a chemical solution can automatically be injected into spike holes of railroad ties during track maintenance. It may also detect the presence of a tie as part of an indexing operation that is a precursor to spike hole filling operation. The present invention could similarly be used to locate ties, tie plates and other features components associated with a railway. The invention also relates to railroad maintenance machines having such a detector system and to a method of using such a system.

Rail anchors, used to secure a rail to railroad ties, typically are held in place by spikes driven into underlying ties that run perpendicular to the rails. These spikes are removed during a variety of maintenance operations, such as a rail re-lay, anchor or tie plate replacement, etc. As a result of the pulling of the spikes that hold the rails to their plates, several holes remain in the tie at the location vacated by the plate. It is usually desirable to fill or "plug" these "spike holes" to prevent rot and water freezing in the open spike holes, which can cause damage to the tie. In addition, should a spike be inserted into an existing spike hole, something of a substance should be in the location to retain the hold-down force of the spike within the tie.

The classic approach to filling spike holes was simply to manually insert cedar plugs into the holes as part of the rail re-lay operation. These plugs initially were inserted by laborers walking along the railway. Later, machines were developed that allowed operators to insert plugs using hand-held tools.

More recently, several different chemical solutions have been developed that are injected into the holes and then react either with a component of the injected material, chemical, or water to form a relatively hard substance that approximates the physical characteristics of wood. Examples of such solutions include a polyurethane-based chemical, an epoxy-based chemical, and a water-based chemical. The first way of injecting these materials was to manually inject the solution into the spike holes using a caulk gun type device or "gun" that simultaneously mixes the constituent chemicals of the solution and injects the solution into the spike holes. This technique is still in use, but generally is limited to relatively small-scale applications such as replacing a short section of railway.

Vehicles have been developed permitting riding operators to manually inject solution into spike holes using guns of the type historically used by walking operators but supplied with chemicals via one or more on-board tanks rather than a self-contained cartridge on the gun. The machine may be either self-propelled and move along the rails or mounted on the back of a pickup truck or the like. These machines typically include a single gun that is manually directed and activated by an operator. Other than being transported by a vehicle and having tanks, these types of devices are, in essence, the same as the traditional caulk gun style operation.

In all of these machines, the guns are controlled, manipulated and triggered by operator rather than being mounted on a work head and operated automatically. In addition, each of these prior machines or techniques required a dedicated operator to each gun rather than permitting a single operator to operate multiple guns.

A number of the drawbacks experienced with previous systems were largely alleviated with the introduction of the improved spike hole filling machine and method described in <CIT> ("the '<NUM> patent"), which is assigned to the assignee of the present application. This machine permits a riding operator to inject a hole filling solution into spike holes as the machine is propelled along the tracks. The injector is provided on a movable workhead so as to be movable multi-axially by the operator using joysticks and other controls to align the injector with the spike holes and inject solution. This machine represents a dramatic improvement over earlier machines in terms of both accuracy and speed.

Nevertheless, further improvements are desired. Hole location and filling operations are still controlled wholly by an operator in machine described in the '<NUM> patent. This requires a skilled operator on the machine, and still risks operator fatigue and error. It is desirable to detect spike holes and automatically align the machine with the holes, after which the holes are filled using various injection materials.

In addition to detection of spike holes, systems are desired that are capable of detecting other railway features, including but not limited to spikes, anchors, and tie plates. Such systems may also allow for automated removal, replacement, and/or repositioning of these components.

Examples of prior art can be found in documents <CIT> and <CIT>.

In accordance with an aspect of the invention, a railroad tie maintenance vehicle for identifying and automatically sealing spike holes in a railroad tie includes a machine that moves along a railway having exposed spike holes in railroad ties. The machine includes an imaging device facing downwardly towards the railroad ties, and at least one injection device that is located adjacent to the imaging device and that is movable vertically and horizontally relative to the tie. A controller is coupled to the imaging device and the injection device and is operable, using information received from the imaging device, to identify the existence and locations of spike holes in the ties and to control the injection device to align an injector thereof with the identified spike holes and to dispense a chemical solution into identified spike holes.

The machine may include a spike hole detecting module, and an execution module. The detecting module includes instructions that, when executed, compare an image captured by the imaging device to a database of images of spike holes. The database may be prestored on the machine and/or in an external server, and may be either static or dynamic and updated with newly acquired images during a spike hole filling operation. The execution module includes instructions that, when executed, move the at least one injection device to be inserted into a spike hole and deposit the chemical solution into the spike hole.

In accordance with another aspect, the machine includes controls that drive the at least one injection device to move relative to the railway to position the at least one injection device in a longitudinal and lateral location proximate to the spike holes without operator input. For instance, a plurality of actuators may be provided that move the at least one injection device in longitudinal and lateral directions, as well as vertically, relative to the vehicle.

According to another aspect, the imaging device includes a camera, a hood that overlies the camera, as well as a light that is located adjacent to the camera and that illuminates the area in the field of view of the camera. The injection device is located directly adjacent to the hood, with a fixed lateral distance between the injection device and the imaging device. Thus, the hood is moved in the lateral direction once a spike hole is identified so that the injection device is located at the identified spike hole along the lateral axis. Further, a longitudinal actuator is configured to move the at least one injection device along the longitudinal axis so that the at least one injection device is located at the identified spike hole along the longitudinal axis.

The machine may have a mothership vehicle and a shuttle cart. The mothership vehicle includes a chassis and a plurality of wheels that support the chassis and that are configured to engage at least one rail. The shuttle cart is movably connected to the mothership vehicle by a physical or wireless tether. The shuttle cart also includes a chassis, as well as a propulsion assembly that is configured to propel the shuttle cart towards and away from the mothership vehicle. The injection device and imaging device are mounted on the shuttle cart.

The mothership vehicle may be configured to move at a substantially constant speed along a rail, whereas the shuttle cart travels at a varying speed along the rail relative to the mothership vehicle. More specifically, the shuttle cart frequently indexes to the next tie in the line, where it stops to fill spike holes, after which it moves along the rail to the next located railroad tie.

In accordance with another aspect, a method of operating a railroad tie maintenance machine is provided. The method includes moving the machine along a railway, capturing images of a plurality of railroad ties using an imaging device on the machine, using a controller on the machine to detect and identify at least one spike hole contained in a given railroad tie, under control of the controller, automatically moving an injection device on the machine to overlie the identified spike hole and dispensing a chemical solution into the spike hole. The moving of the machine may include moving a mothership vehicle along the railway and moving a shuttle cart relative to the mothership vehicle, and the shuttle cart supports the imaging device and the injecting device. The moving of the injection device may comprise actuating a plurality of actuators to move the injection device in lateral and longitudinal directions, and then vertically towards and then away from the spike holes. The controller may include or be in communication with one or more databases of images of spike holes, which may be static or may be dynamic and updated during spike hole filling operation. In addition to identifying spike holes, the controller may also be configured to detect other aspects of a railroad tie, such as imperfections in the tie, cracks in the tie, ballast rock or other foreign objects on the tie.

These and other aspects and objects of the present invention will be better appreciated and understood when considered in conjunction with the following description and the accompanying drawings. It should be understood, however, that the following description, while indicating preferred embodiments of the present invention, is given by way of illustration and not of limitation. Many changes and modifications may be made within the scope of the present invention, and the invention includes all such modifications.

A clear conception of the advantages and features constituting the present invention, and of the construction and operation of typical mechanisms provided with the present invention, will become more readily apparent by referring to the exemplary, and therefore non-limiting, embodiments illustrated in the drawings accompanying and forming a part of this specification, wherein like reference numerals designate the same elements in the several views, and in which:.

In describing the preferred embodiment of the invention which is illustrated in the drawings, specific terminology will be resorted to for the sake of clarity. However, it is not intended that the invention be limited to the specific terms so selected, and it is to be understood that each specific term includes all technical equivalents which operate in a similar manner to accomplish a similar purpose. For example, the words "connected", "attached", "supported", or terms similar thereto are often used.

The present invention and the various features and advantageous details thereof are explained more fully with reference to the non-limiting embodiments described in detail in the following description.

In this specification, the following non-SI units are used, which may be converted to the respective SI or metric unit according to the following conversion table :.

Referring to the general to the drawings, an automated tie filling machine <NUM> or vehicle for filling spike holes <NUM> in railroad ties <NUM> of a railway <NUM> is illustrated.

Initially, the machine <NUM> will be described in connection with automated filling of spike holes <NUM> using a visual detection system that acquires still or video images of the spike holes <NUM>. Additional examples will be provided below in which the visual detection system is used to detect other components associated with a railway.

The machine <NUM> typically will be used as part of a track re-laying operation in which a rail <NUM> is removed from the railway and replaced with a new rail. As part of this process, the spikes <NUM> holding the rail <NUM> to the tie <NUM> and holding the tie plate <NUM> to the rail <NUM> are removed by one or more machines (not shown) working ahead of the tie filling machine <NUM>, leaving holes <NUM> in the ties <NUM>, and one of the rails (the right rail in the illustrated example) is removed. Those holes <NUM> should be filled to preserve the integrity of the tie <NUM> and to provide solid surfaces into which new spikes (not shown) may be driven. Filling typically involves the injection of a chemical solution <NUM> (<FIG>) into the holes <NUM>, the solution <NUM> being formed from a two-part epoxy, polyurethane, or a water-based system.

Turning initially to <FIG>, the tie filling machine <NUM> is a self-propelled machine that moves along a railway <NUM>. The machine <NUM> of this embodiment includes a mothership vehicle <NUM> that is propelled along the railroad <NUM>, as well as a shuttle cart <NUM> that is a physically separate module from the mothership vehicle <NUM>. The shuttle cart <NUM> is movable relative to the mothership vehicle <NUM>, and is physically and/or wirelessly tethered to the mothership vehicle <NUM> as will further be described below. The mothership vehicle <NUM> has a chassis <NUM> that is supported on the remaining left rail <NUM> of the railway <NUM> on at least one set of wheels <NUM>. The mothership vehicle <NUM> also includes one or more crawlers <NUM>, which are also provided for supporting the machine <NUM> on the railbed <NUM> in the vicinity of the removed right rail, permitting propulsion along railway sections in which one rail <NUM> has been removed. Each crawler <NUM> is mounted on a mast <NUM> so as to be movable vertically into and out of engagement with the railbed <NUM> via operation of an associated hydraulic cylinder <NUM>. The mothership vehicle <NUM> may also be configured with crawlers <NUM> on both sides of its chassis <NUM>, allowing operation with both rails <NUM> removed from the track. The mothership vehicle <NUM> is propelled by a motive power source <NUM>, such as a gasoline or diesel engine.

Still referring to <FIG>, the mothership vehicle <NUM> includes an operator workstation <NUM> which, in this embodiment, is enclosed cab. Operation of the mothership vehicle <NUM>, as well as some of the operations of the shuttle cart <NUM>, are controlled by an operator located in the operator workstation <NUM>. The operator workstation <NUM> is located generally centrally of the chassis <NUM> adjacent the front end of the mothership vehicle <NUM> where the shuttle cart <NUM> is movable relative to the mothership vehicle <NUM>. The operator workstation <NUM> preferably takes the form of an environmentally controlled enclosed cab that shields a seated operator within from the elements. Windows <NUM> may be provided on various sides of the operator workstation <NUM> to allow the operator to direct movement of the tie filling machine <NUM>, including the mothership vehicle <NUM>, the shuttle cart <NUM>, as well as any associated components. The operator workstation <NUM> may include multiple controllers, displays, touch screens, and the like referred to collectively as user interface <NUM> in <FIG>, that enables the operator to effectively operate the machine. For instance, the operator may use the controllers, displays, or touch screens to change any number of operating characteristics of the mothership vehicle <NUM> and the shuttle cart <NUM> described herein. These devices will henceforth collectively and individually be referred to as "interfaces" for the sake of conciseness. Further, the positions of the various moving components of the machine <NUM> may be monitored by the operator using fixed displays such as gauges or dials and/or variable displays such as touch screens. The status of any or all of the components of the machine could similarly be monitored by any combination of various electrical sensors, optical sensors, limit switches, etc. (not shown) and displayed to the operator via any desired combination of fixed or variable displays. Similarly, some of these components, or even the entire machine, could similarly be monitored or even controlled from a remote location using an interface that communicates wirelessly with the controller. The operator workstation <NUM> could be eliminated in this latter case.

The shuttle cart <NUM> will now be further described with reference to <FIG>. The shuttle cart <NUM> travels along the railroad to plug or fill the spike holes <NUM>, and is movable relative to the mothership vehicle <NUM>. The shuttle cart <NUM> includes a chassis <NUM> and at least one propulsion device that enables separate propulsion of the shuttle cart <NUM> along the railbed <NUM> relative to the mothership vehicle <NUM>. The propulsion device takes the form of a crawler <NUM> in this embodiment. The crawler <NUM> is similar in operation to the crawler <NUM> associated with the mothership vehicle <NUM>. The crawler <NUM> is located on a working side of the shuttle cart <NUM>, meaning the side where a rail has been removed and the spike holes <NUM> are being filled. Of course, another crawler may be provided on the opposite side of the shuttle cart in an application in which both rails have been removed.

Additionally, the shuttle cart <NUM> includes first and second wheels <NUM> that rollably support the shuttle cart <NUM> on the rail <NUM> on the non-working side of the shuttle cart <NUM> opposite of the working side having the crawler <NUM>. Further still, rail clamps <NUM> having first and second fingers or claws <NUM> may be positioned directly adjacent to each of the wheel guides <NUM>. The rail clamps <NUM> are configured to be releasably adjusted between an opened position in which the first and second fingers <NUM> are spaced from the rail <NUM>, and a closed position in which the first and second fingers <NUM> help secure the shuttle cart <NUM> to the rail <NUM>.

In operation, the mothership vehicle <NUM> is typically propelled along the railroad at a substantially constant speed by its crawlers <NUM>, while the shuttle cart <NUM> is able to move back and forth relative to the mothership vehicle <NUM> at different speeds by its crawler <NUM>. The crawler <NUM> is powered by a hydraulic motor <NUM> using pressurized fluid supplied by the mothership vehicle <NUM>, although other motors could similar be used to facilitate operation of the crawler <NUM>. By allowing the shuttle cart <NUM> to be moveable relative to the mothership vehicle <NUM>, the mothership vehicle <NUM> can move at a substantially constant speed, rather than starting and stopping every time a tie <NUM> is filled, which would result in frequent momentum changes to the mothership vehicle <NUM>, and in turn the operator located therein. Of course, when the shuttle cart <NUM> is not filling spike holes <NUM>, the mothership vehicle <NUM> and the shuttle cart <NUM> can move together along the railway at a substantially constant speed. When this occurs, the mothership vehicle <NUM> and the shuttle cart <NUM> can be mechanically coupled to one another.

Turning back to <FIG> and <FIG>, the shuttle cart <NUM> is connected to the mothership vehicle <NUM> by a tether <NUM> composed of a plurality of hoses <NUM> and cables. Alternatively, the shuttle cart <NUM> can be connected to the mothership vehicle <NUM> by a boom (not shown) containing the hoses <NUM>. These hoses and cables <NUM> contain a variety of instruments and materials that connect or are transported between the mothership vehicle <NUM> and the shuttle cart <NUM>. For instance, the hoses and cables <NUM> may include one or more of electrical wires or cables to transfer power, as well as instructions and data, to and from the mothership vehicle <NUM> to the shuttle cart <NUM>, as well as chemical filling solution <NUM> or hydraulic fluid transported from the mothership vehicle <NUM> to the shuttle cart <NUM>. It is conceivable that at least some aspects of the tether need not be physical. For example, electrical signals could be transmitted wirelessly between the mothership vehicle <NUM> and the shuttle cart via a RF, Bluetooth®, or other remote connection. Alternatively, the tether could form a more robust structure, such as an extendable boom mechanically coupling the shuttle cart <NUM> to the mothership vehicle <NUM> by the boom (not shown).

It also should be noted that that a separate mothership vehicle <NUM> and shuttle cart <NUM> are not essential to the use of vision to identify, locate, and manipulate railway features. It is conceivable that the shuttle cart <NUM> could be eliminated in favor of one or more traditional workhead(s) on a vehicle that has the other characteristics of the mothership vehicle <NUM> disclosed herein. In this case, a portion or potentially all of the longitudinal movement of the components carried by the shuttle cart <NUM> of the illustrated embodiment could be mounted on a workhead that is moveably longitudinally of a frame on the vehicle. An example of such longitudinally movable workhead is disclosed in the '<NUM> patent.

Returning again to the illustrated embodiment, the shuttle cart <NUM> includes one or more filler workheads <NUM> that are movable relative to the chassis <NUM>, as best seen in <FIG>. The filler workheads <NUM> are configured to identify spike holes <NUM>, determine the location of the spike holes <NUM> on the railbed <NUM>, and detect the presence of spike holes <NUM> and to automatically fill the identified spike holes <NUM>. First and second filler workheads 66A and 66B are provided one in front of the other in the illustrated embodiment. Providing two filler workheads 66A, 66B permits the machine <NUM> to act on two consecutive ties <NUM> simultaneously, increasing the workpace of the machine <NUM>. The description of the common reference character "<NUM>" as used henceforth will apply equally to both filler workheads 66A and 66B unless otherwise specified.

As best seen in <FIG>, the filler workhead <NUM> includes a frame <NUM> mounted on rollers <NUM> that move along rails <NUM> extending laterally of the railway. The frame <NUM> supports an imaging system <NUM> and an injection wand assembly <NUM> configured to inject chemical solution <NUM> into spike holes <NUM> formed in the ties as well as a hood <NUM>. A hydraulic cylinder <NUM> is operable to drive the frame <NUM> along the rails <NUM>.

The injection wand assembly <NUM> is best shown in <FIG> and includes frame <NUM> fixedly mounted on the frame <NUM> of the filler workhead <NUM>. The frame <NUM> is mounted on longitudinally extending guide rails <NUM> so as to movable back and forth in the longitudinal direction <NUM>, as best seen in <FIG> and <FIG>. This back and forth movement occurs under action of a belt <NUM> that is driven by a pully <NUM> that, in turn, is driven by a hydraulic motor <NUM>. Still referring to <FIG>, the injection wand assembly <NUM> further includes a mixing tube <NUM>, a mixing tube shield <NUM> surrounding the mixing tube <NUM>, and various valves <NUM> configured to control flow of selected amounts of chemicals into the mixing tube <NUM>. As shown, the valves <NUM> include manually operated shut-off valves that can be closed, for example, during maintenance to prevent the inadvertent leaking of chemicals down the injection wand assembly <NUM>. The mixing tube <NUM> extends downwardly from the frame <NUM> and terminates in a tip <NUM> having a lower injection nozzle or orifice. One or more chemicals are deposited into the mixing tube <NUM> in desired proportions by the valves <NUM> and then mixed in preparation for deposit of the solution <NUM> into identified spike holes <NUM>. The mixing tube shield <NUM> surrounds a portion of the mixing tube <NUM> to prevent inadvertent contact of the mixing tube <NUM> by other components.

Still referring to <FIG>, the injection wand assembly <NUM> can be driven to move vertically relative to the frame <NUM> and, thus, relative to the frame <NUM>, by a Z actuator. In the present case, the Z actuator is a pneumatic cylinder <NUM> that extends and retracts in the vertical direction <NUM>. Relieve valves <NUM> are provided at either end of the cylinder provided with relief valves <NUM> located at either end of the cylinder <NUM> in order to expedite movement of the injection wand assembly <NUM> in the vertical direction. Additionally, vertical guide rods <NUM> may be mounted adjacent to the cylinder <NUM> to guide the injection wand assembly <NUM> during its vertical movement relative to the shuttle cart <NUM>. Further, springs <NUM> may be mounted adjacent to the cylinder <NUM> in order to bias the injection wand assembly <NUM> away from the tie <NUM> once the spike hole <NUM> has been filled. The springs <NUM> bias the cylinder <NUM> away from the spike hole <NUM> so that the injection wand assembly <NUM> is withdrawn from the spike hole <NUM> even if air supply to the air cylinder <NUM> is terminated so that the tip <NUM> is not stuck within the spike hole <NUM> even if the machine is powered down or air pressure otherwise is lost.

Referring again to <FIG>, the hood <NUM> is located directly adjacent to the injection wand assembly <NUM>, and overlies the imaging device <NUM>. The imaging device <NUM> of this embodiment is a camera <NUM> facing downwardly from the shuttle cart <NUM> to a tie <NUM> and at least one light <NUM> configured to illuminate the tie <NUM> while the camera <NUM> captures still or video images of the tie <NUM> and surrounding features as identified by the field of view FOV identified in <FIG>. More specifically, the camera <NUM> in the illustrated embodiment is a BASLER camera model Ace U acA720-290gc camera, equipped with <NUM> millimeter wide-angle C-mounted lens. Of course, other cameras and lenses having similar characteristics could similarly be used with the present machine <NUM>. The images are then analyzed using computer vision (CV) software to identify the specific location of each tie <NUM> and of spike holes <NUM> in each tie <NUM> using artificial intelligence (AI). The CV software could be located within the controller <NUM> and/or located in cloud-based external server in wireless communication with the controller <NUM>.

In one embodiment, the CV software uses Tensorflow software, which is an open-source software library for machine learning and artificial intelligence that can be used across a range of tasks, with a focus on training and inference of deep neural networks. The Tensorflow software includes two stages: training and inference. In the training stage, a large amount of data is collected and stored. In the present invention, the data is images associated with specific portions of a railway, for instance, of ties and holes formed in the ties. This data is annotated and labeled using a labelling software to draw boxes around the ties and holes and label them respectively. Once the images have been labeled, the Tensorflow software then develops a model based on the provided data. In the present embodiment, an object detection model is generated, which is used to identify all instances of a particular object within a single image. The model is essentially an equation or process where an input type is entered (in the present invention, an image) and the software outputs the coordinates and confidence score of items detected in the image. The model is then shared across the machine <NUM>. The CV software is configured to continuously refine the training process, such that the model is constantly updated to improve performance. The training stage may be started from scratch, or it can be built off of a pre-trained model. In one embodiment, a pre-trained Fastere-RNN model is used that begins the ability to perform object detection, which is further refined based on the objects being identified.

In the inference stage, the model that was previously created during the training stage is actually put to use. As will further be described below, an image is acquired from the imaging device <NUM> and is fed into the model. The model then identifies a specific type of item, such as a railroad tie <NUM> or a spike hole <NUM>, and a confidence score is calculated. This information is transmitted to the controller <NUM>, and various components of the machine <NUM> are appropriately moved and manipulated as will be described below.

Identification and filing of spike holes <NUM> will now be described.

In operation, the shuttle cart <NUM> is configured to move along the railway <NUM> and the mothership vehicle <NUM> as the mothership vehicle <NUM> is independently propelled along the railway <NUM> while images are acquired via the camera. The images may be still images and/or video frames. The acquired images are then analyzed by the CV software, against a model including a database of photographs of previously identified spike holes <NUM>. As described, the model may be dynamic and be updated as the machine <NUM> moves along the railway <NUM>, otherwise it may be static. Initially, the tie <NUM> is identified and images of the tie <NUM> are taken by the camera <NUM>. The images are then transmitted to the controller <NUM> and/or to a remote server for analysis. In either event, the images are compared to the model including a database of images using the CV software described above in order to identify the existence and location of the ties <NUM> as well as the existence and location of the spike holes <NUM> in the ties <NUM>. In many instances, a pattern of four spike holes <NUM> will be identified, although the CV software can similarly identify any number of spike holes <NUM> that may be present in a given tie <NUM>.

While analyzing the images, the CV software is configured to specifically identify spike holes <NUM> as such, while also disqualifying other railway features, such as cracks in the tie, imperfections in the tie such as knots, stains, ballast rocks or other foreign objects located on a tie <NUM>, and the like. <FIG> shows a sample tie <NUM> that includes spike holes <NUM> along with a crack "C" and a stain "S", where the CV software differentiates the various features. Furthermore, the CV software may be configured to toggle between different modes in which characteristics such as color may differ, for instance, to reflect changes in conditions of the tie <NUM> such as the presence of a foreign object, including snow or rain on top of the tie <NUM>. To do so, the CV software compares images taken by the camera <NUM> to a database of images in snowy conditions or a database of photographs in rainy conditions. The system may continue to collect and analyze all images taken, dynamically updating the database with new images, which further improves the accuracy of the CV software.

After the spike hole <NUM> locations have been confirmed, the cylinder <NUM> is driven to move the frame <NUM> along the rails <NUM> laterally relatively to the shuttle cart chassis <NUM> as shown by arrow <NUM>, and motor <NUM> can be actuated to drive the belt <NUM> to move the injection wand assembly <NUM> longitudinally of the frame <NUM> as shown by arrow <NUM> until the injection wand assembly <NUM> directly overlies an identified spike hole <NUM>. As can be seen in <FIG>, the cylinders <NUM> then can be actuated to move the injection wand assemblies <NUM> vertically, as shown by arrow <NUM>, to fill the spike hole. Specifically, the tip <NUM> of the tube <NUM> is lowered into a first detected spike hole <NUM>, filled with solution <NUM> and then raised, after which the position of the wand assembly <NUM> is adjusted laterally and longitudinally to position the tip <NUM> of the tube <NUM> over each subsequently detected spike hole <NUM>. The tip <NUM> is preferably inserted at a distance far enough into the spike hole <NUM> to ensure the filling solution <NUM> is deposited within the spike hole <NUM>, without going so far into the spike hole <NUM> as to cause splashing within the spike hole <NUM> or otherwise prevent the spike hole <NUM> from being fully filled. For instance, in a preferred embodiment, the tip <NUM> is lowered into a detected spike hole <NUM> to approximately a quarter of an inch beneath the top surface of the tie <NUM>. Of course, depending on the characteristics of the detected spike holes <NUM>, the tip <NUM> may be inserted into the spike hole <NUM> at a distance greater than or less than a quarter of an inch. For instance, where the spike holes <NUM> for a particular set of ties <NUM> are deeper than typical spike holes, the tip <NUM> may be inserted at a distance deeper than a quarter of an inch. To the contrary, where spike holes <NUM> for a particular set of ties <NUM> are shallower than typical spike holes, the tip <NUM> may be inserted at a distance less than a quarter of an inch. The operator may set the insertion depth of the tip <NUM> by an interface connected in the cabin <NUM> in communication with the controller <NUM>.

The injection wand assembly <NUM> receives chemical solution from one or more storage tanks. In the illustrated embodiment, a solution <NUM> is a two-part resin solution. The two components are stored in large storage tanks <NUM>, <NUM>, on the mothership vehicle <NUM> and then transferred to the shuttle cart <NUM> through one or more of the hoses <NUM> of the tether <NUM>. Of course, storage tanks of the solution could similarly be mounted on the shuttle cart <NUM>. The hoses <NUM> may be housed within a hose protector <NUM> that extends laterally along the chassis <NUM> of the shuttle cart <NUM>. The chemicals are then delivered to the mixing tube <NUM> through additional hoses <NUM> under the control of valves <NUM> that are automatically actuated by the controller <NUM>. Each tank <NUM>, <NUM> preferably has a sufficient capacity to permit continuous operation of the shuttle cart <NUM> for an extended period of time. A per-tank capacity of at least <NUM> gallons, and more preferably at least <NUM> gallons, is typical.

The solution <NUM> is typically deposited into the spike holes <NUM> for between approximately <NUM>-<NUM>,<NUM> milliseconds, and more typically approximately <NUM> milliseconds depending on a number of factors, including for instance the viscosity of the material, the temperature of the material and outdoor conditions, and the size of the spike hole <NUM>. About <NUM> to <NUM> inches<NUM>, or between <NUM> and <NUM> milliliters, of solution, and more preferably typically approximately <NUM> inches<NUM> or <NUM> milliliters of solution, are deposited within each spike hole <NUM> as a result of this process. Of course, an operator can use the interface to adjust the length of time that the filling solution is sprayed from the injection wand assembly <NUM> and the quantity of filling solution <NUM> that is deposited into the spike hole <NUM>. In optimized conditions, the machine <NUM> preferably fills the spike holes associated with <NUM>-<NUM> ties per minute.

The machine <NUM> may also be equipped with additional components to ensure proper interaction between the mothership vehicle <NUM> and the shuttle cart <NUM>. For instance, mothership vehicle <NUM> may include a sensor (not shown) that monitors the distance between the mothership vehicle <NUM> and the shuttle cart <NUM>. Similarly, a rangefinder (not shown) may be used that measures the distance between the mothership <NUM> and the shuttle cart <NUM>. Further still, the distance between the mothership vehicle <NUM> and shuttle cart <NUM> may also be monitored using a physical leash (not shown) that helps the operator monitor the distance between these components, and terminate movement if the measured distance surpasses a designated amount. Whether a sensor, rangefinder, or physical leash is used, machine operation can be controlled so that a maximum permissible distance, for instance between <NUM>' and <NUM>' and, more typically, between <NUM>' and <NUM>', between the mothership vehicle <NUM> and shuttle cart <NUM> cannot be exceeded. Additionally, the shuttle cart <NUM> may include safety bars <NUM>, <NUM> on either end of the shuttle cart <NUM>. Both safety bars <NUM>, <NUM> are pivotable about the chassis <NUM> of the shuttle cart <NUM>. When either bar <NUM>, <NUM> is pivoted due to contact with another object, propulsion of the shuttle cart <NUM> can be terminated, and the operator can be notified to ensure appropriate safety measures are in place.

Turning to <FIG>, the various components of the system <NUM> for identifying and filling spike holes <NUM> are illustrated. Many of the components described above are shown in this schematic, including the controller <NUM>, the imaging system <NUM> including the camera <NUM> and a processor, the user interface <NUM>, the cylinder <NUM> that accounts for lateral motion of the imaging system <NUM>, the belt <NUM> that accounts for longitudinal motion of the imaging system <NUM>, the cylinder <NUM> that accounts for vertical motion of the imaging system <NUM>, and the injection wand assembly <NUM>. Further, a shuttle cart propulsion system <NUM> that includes the crawlers <NUM> is also provided. Other sensors <NUM> may also transmit data and information to the controller <NUM>. The imaging system <NUM> and user interface <NUM> are configured to both transmit data and information to the controller <NUM>, as well as receive data and information from the controller <NUM>. Similarly, the cylinder <NUM> that accounts for lateral motion of the imaging system <NUM>, the belt <NUM> that accounts for longitudinal motion of the imaging system <NUM>, the cylinder <NUM> that accounts for vertical motion of the imaging system <NUM>, and the injection wand assembly <NUM> both receive data and information from the controller <NUM>, as well as transmit information and data to the controller <NUM> to confirm that the injection wand assembly <NUM> has reached a desired location. The controller <NUM> is configured to push data and information to the shuttle cart propulsion system <NUM>, as well as the injection wand assembly <NUM>.

Turning next to <FIG>, <FIG>, and <FIG>, exemplary flow diagrams are provided to further illustrate operation of the machine <NUM>. Looking to <FIG>, a first flow chart illustrates the initialization process at block <NUM> associated with the CV software that occurs at the processor of the imaging system <NUM> when the injection wand assembly <NUM> is located over a tie <NUM>. Next, at a determination step <NUM>, a camera <NUM> is designated as the lead by the operator. For instance, depending on whether the shuttle cart <NUM> is filling spike holes <NUM> on the left or the right side of the railway <NUM>, one of the two cameras <NUM> will be the lead, whereas the other of the two cameras <NUM> will be lead if the shuttle cart <NUM> is filling spike holes <NUM> on the right-side of the railway. Next, an inquiry is made at block <NUM> as to whether a hole locations request has been received from the CV software. If no hole location request has been received, the lead camera <NUM> acquires a video or still image at block <NUM>. The image is then processed by the model of the CV software, and the location of the tie <NUM> is extracted at block <NUM> once the tie is recognized or identified. Thereafter, the coordinates of the tie <NUM> are transmitted to the controller <NUM> by a CAN channel at block <NUM>. Once the hole locations request is received, images are acquired by both cameras <NUM> at block <NUM>. If it is determined in block <NUM> that a hole location request has been received, images are acquired in block <NUM>, and the process proceeds to block <NUM>, where spike holes <NUM> are identified as such using the model, and the locations of the identified spike holes <NUM> are assigned to a coordinate system in terms of lateral and longitudinal ("X" and "Y"). The coordinates are then sent over CAN to the controller <NUM> at block <NUM>. At block <NUM>, a determination is made as to whether work is complete, and the signal/coordinates have been received. If the answer is "no", the determination is repeated. If the answer is "yes", the hole locations step at block <NUM> is repeated.

Looking to <FIG>, a second flow chart is provided that illustrates the processing of images from the camera <NUM> by the processor of the imaging system <NUM>. Initially, a frame is acquired from the camera at block <NUM>. Once acquired, the frame is converted to appropriate format and size at block <NUM>. Additionally, any lens distortion of the image is removed at block <NUM>. Next, the image is fed into the model, and the model performs the inference step at block <NUM>, in which the model detects a spike hole or holes <NUM>. After that, the inference results are sorted for relevant data at step <NUM>. The relevant data is then formatted for the controller <NUM> at step <NUM>. Finally, the formatted data is sent to the controller <NUM> over the CAN at step <NUM>.

Next, <FIG> shows another simplified flow diagram of the process of controlling the machine <NUM> by the controller <NUM>, working with the imaging system <NUM>, to identify, locate, and fill spike holes <NUM>. At block <NUM>, connection is initiated by the camera <NUM> and the CV software as discussed above in connection with <FIG>, after which media is transmitted to the controller <NUM> over the CAN at step <NUM>. Next, the shuttle cart <NUM> is advanced longitudinally along the rail at block <NUM> by driving the crawler <NUM>. Then, at block <NUM>, the CV software determines whether a tie <NUM> has been detected. If not, the workhead cart is advanced along the railway to continue to search for a tie. If so, the shuttle cart stops over the tie. This moving along the rail to position the imaging device over the next tie of the railway can be considered "indexing". The tie <NUM> is analyzed by the CV software at block <NUM>. Once the tie is detected and the imaging device is indexed over it in block <NUM>, the CV software then determines the presence and location of any spike holes <NUM>. If any spike holes are located, the injection wand assembly <NUM> is moved in a lateral and longitudinal direction under power of the cylinder <NUM> and the motor <NUM> in combination with the belt <NUM>, respectively, until the tube <NUM> of the injection wand assembly <NUM> is positioned over the spike hole <NUM> at block <NUM>. Once the injection wand assembly <NUM> is properly positioned in block <NUM>, the cylinder <NUM> is actuated to lower the injection wand assembly <NUM> to insert the tube tip <NUM> into a first spike hole <NUM> at block <NUM>, and a chemical solution <NUM> is injected into the spike hole <NUM> at block <NUM>. The wand assembly <NUM> then is raised to lift the tube tip out of the spike hole. An inquiry is then made in block <NUM> as to whether all of the spike holes on the subject tie have been filled. If not, the process proceeds to block <NUM> and the injection wand assembly <NUM> is repositioned in X and Y over a subsequent spike hole <NUM> at block <NUM>. The injection wand assembly <NUM> is then lowered to place the tube tip <NUM> into the subsequent spike hole <NUM> at block <NUM>, and the solution <NUM> is injected into the spike hole at <NUM>. The wand assembly <NUM> then is raised again, and the process returns to the inquiry block <NUM>. Once it is determined in inquiry block <NUM> that all spike holes <NUM> of a given tie <NUM> have been filled, the process returns to block <NUM>, and the shuttle cart <NUM> is advanced along the track to index the imaging system over the next tie <NUM>.

While the above description relates to filling spike holes <NUM> in a single tie <NUM>, since there are two cameras <NUM>, injection wand assemblies <NUM>, and associated components, spike holes <NUM> in two ties <NUM> could be filled simultaneously by the first and second injection wand assemblies.

As described above, while the detailed description primarily discusses use of the machine <NUM> to identify the location of spike holes <NUM>, the vision system described herein, or contemplated variations of it, could similarly be used to visually identify, locate, and act on other railway features. Such features include, but are not limited to, tie plates, spikes, rail anchors, and rail clips. The tool could act upon the feature by one or more of removing it, placing it, attaching it (such as by driving a spike or applying an anchor or clip), removing it (such as by pulling a spike or "spreading" anchors), etc..

For instance, <FIG>, provides a schematic of one such system <NUM> in which the machine can be used to identify, locate move, and/or replace a tie plate. This system <NUM> includes a number of components that operate substantially similarly to those described in system <NUM>, including a controller <NUM>, an imaging system <NUM> including a camera (not shown) and processor, a user interface <NUM>, and propulsion system <NUM>, as well as other sensors <NUM>. If the system has a mothership vehicle and a shuttle cart, the propulsion system will control both vehicles as described above. If the system has a single vehicle with a workhead movable longitudinally along a frame of the vehicle, the drive for effecting that longitudinal movement can be considered part of the propulsion system. Additionally, the system <NUM> includes a tool <NUM> that is configured to move a tie plate <NUM> (see <FIG>) along the tie <NUM> laterally of the railway. Additionally, the system <NUM> includes multiple actuators to facilitate movement of the tool <NUM> relative to the railway, including a lateral actuator <NUM> to facilitate movement along the x-axis, a longitudinal actuator <NUM> to facilitate movement along the y-axis, and a vertical actuator <NUM> to facilitate movement along the z-axis. The imaging system <NUM>, user interface <NUM>, lateral actuator <NUM>, longitudinal actuator <NUM>, and vertical actuator <NUM> are all configured to both transmit data and information to the controller <NUM>, as well as receive data and information from the controller <NUM>. The controller <NUM> is configured to control operation of the propulsion system <NUM>, as well as the actuators for the tool <NUM>.

Although the best mode contemplated by the inventors of carrying out the present invention is disclosed above, practice of the present invention is not limited thereto. It will be manifest that various additions, modifications and rearrangements of the features of the present invention may be made without deviating from the scope of the underlying inventive concept.

Moreover, the individual components need not be formed in the disclosed shapes, or assembled in the disclosed configuration, but could be provided in virtually any shape, and assembled in virtually any configuration. Furthermore, all the disclosed features of each disclosed embodiment can be combined with, or substituted for, the disclosed features of every other disclosed embodiment except where such features are mutually exclusive.

Claim 1:
A railway maintenance machine comprising:
a chassis (<NUM>), the chassis (<NUM>) being configured for movement along a railway (<NUM>);
an imaging device (<NUM>) which faces downwardly towards the railway (<NUM>) and which is configured to acquire images of the railway (<NUM>);
an injection device (<NUM>) that is associated with the imaging device (<NUM>) and that is configured to fill spike holes (<NUM>) in a railroad tie (<NUM>); and
a controller (<NUM>) that is coupled to the imaging device (<NUM>) and the injection device (<NUM>), the controller (<NUM>) including:
a spike hole detecting module including instructions that, when executed, compare an image captured by the imaging device (<NUM>) to a database of images of spike holes; and
an execution module including instructions that, when executed, automatically move the injection device (<NUM>) into alignment with a detected spike hole (<NUM>) and cause the injection device (<NUM>) to inject a chemical solution (<NUM>) into the spike hole (<NUM>) to fill the spike hole (<NUM>).