Patent ID: 12239059

DETAILED DESCRIPTION

In exemplary embodiments, the present invention is described in the context of vertical farming, but it should be appreciated that one or more of the various components, systems and processes described herein may be applied to other types of agriculture, such as, for example, indoor farming, outdoor farming, greenhouse farming, vertical farming and non-vertical farming, to name a few.

As explained herein, various components of the inventive vertical farm system are stationary while other components are not stationary. In this regard, the term “stationary” should be taken to mean fixed in place as in the case of, for example, manufacturing fixtures that hold a workpiece in a fixed position during a manufacturing process. As a more specific example related to the present invention, robots may be stationary in that they are fixed to a non-moveable platform within the manufacturing environment but are otherwise free to move to carry out manufacturing tasks. In contrast, components that are not stationary are free to move from point to point within the vertical farm system, and are not fixed in place.

FIG.1shows a layout of a vertical farm system, generally designated by reference number1, according to an exemplary embodiment of the present invention. The vertical farm system1includes an enclosure10that houses the main components of the system1. In exemplary embodiments, the enclosure10may be a cleanroom and in order to minimize the carrying of particulate by a person moving into the enclosure10, workers and other personnel may enter and leave through airlocks with or without an air shower stage, and wear protective clothing such as hoods, face masks, gloves, boots, and coveralls. The enclosure10may be a stand-alone structure or part of a facility that includes multiple enclosures10, and in exemplary embodiments may be a walled off-section of a warehouse, a shipping container, or some other generally enclosed room.

The vertical farm system1includes a plurality of racks20that are configured to hold agricultural crops growing within the contained environment provided by the enclosure10. In exemplary embodiments, the racks20are configured to move along a generally rectangular path within the enclosure10, as indicated by arrows A. In this regard, the vertical farm system1may include a conveyor system40on which the racks20are mounted and moved within the enclosure10. In exemplary embodiments, and as explained in more detail below, the conveyor system40may include a track on which the racks20are guided as the racks20move through the enclosure10. The vertical farm system1may include a plurality of enclosures10with corresponding racks20and conveyor systems40, with each enclosure10preferably sealed off from the other enclosures10to prevent cross-contamination.

In exemplary embodiments, the crops grown in the vertical farm system1may be flowering crops, such as, for example, strawberries, tomatoes, melons, peppers, eggplants and berries, to name a few, as well non-flowering crops such as leafy greens, root vegetables and mushrooms, to name a few. Additionally, crops grown may include tree crops, such as citrus, apples, tree nuts and olives, to name a few, as well as staple crops such as wheat, rice and corn, to name a few.

As shown, the enclosure10is divided in half to provide both a day-time cycle and a night-time cycle. As explained in more detail below, during the day-time cycle, lighting is provided to simulate sunlight so as to stimulate growth of the crops, while no lighting is provided in the night-time cycle. In exemplary embodiments, the day and night-time cycles may be based on any number of total hours, such as, for example, 6 hours, 12 hours, 24 hours or more. For example, if the total number of hours in the “day” or photoperiod is 24 hours, the number of hours that make up the day-time cycle might be 12 hours and the number of hours that make up the night-time cycle might be 12 hours, or any other time periods that add up to the total 24-hour photoperiod. It should be appreciated that the number of hours in the “day” is not limited to 24 hours, and in exemplary embodiments the number of hours in each “day” may be less or more than 24 hours, and each “day” may vary in the number of total hours (for example, 22 hours in a first day, 26 hours in a second day, 20 hours in a third day, etc.) In exemplary embodiments, the number of hours of the day-time cycle might be equal or not equal to the number of hours of the night-time cycle. For example, if the number of hours in the “day” is 24 hours, the day-time cycle might be 14 hours and the night-time cycle might be 10 hours. Further, in exemplary embodiments, the number of hours of the day-time cycle and the number of hours of the night-time cycle might vary from day to day.

The two halves of the enclosure10may be divided by a partition12made of, for example, plastic, fabric, metal panels (insulated or not insulated) or any other suitable material, and which is opaque enough to block a substantial amount of the light from entering the night-time portion of the enclosure10. In exemplary embodiments, rather than a partition12, the enclosure10may be divided into separate rooms, with one room providing the day-time cycle and the other room providing the night-time cycle.

The system1further includes a harvesting station500and a worker platform65. As explained in more detail below, the harvesting station500may be a robotic harvesting station that includes one or more robots controlled to harvest ripe or semi-ripe fruit or vegetables as the crop matures. The worker platform65may include components, such as, for example, scaffolding, ladders and lifts, to name a few, to allow workers to access the racks20at varying heights as the racks20pass by the workers. In exemplary embodiments, the harvesting station500and worker platform65are generally stationary compared to the racks20, which again are moved around the enclosure on the conveyor system40. The harvesting station500and the worker platform65may be located at any point within the enclosure10, such as, for example, at each end of the enclosure10, at the middle of the enclosure10, or at the sides of the enclosure10. The harvesting station500and the worker platform65may be positioned directly adjacent to one another at the same location within the enclosure10or spaced from one another at different locations within the enclosure10. In exemplary embodiments, multiple harvesting stations500and/or multiple worker platforms65may be positioned throughout the enclosure10.

The system1also includes an irrigation system made up of one or more irrigation stations70placed at spaced locations along the path of the racks20. As explained in more detail below, the irrigation stations70are generally stationary compared to the racks20, and operate to provide water or water-fertilizer solution (referred to herein as “irrigation fluid”) to the crops held on each rack20and drain the water or water-fertilizer solution from the racks20.

FIGS.2-4show a rack20according to an exemplary embodiment of the present invention. The rack20includes a vertical central frame22that is attached to the conveyor system40and that holds a plurality of horizontally oriented gutters24. Although the rack20is shown having eight gutters24, it should be appreciated that each rack20may include any number of gutters24, such as, for example, four, six, ten, twelve or more gutters24. In exemplary embodiments, the rack20does not include any outer frame elements, and instead the rigidity of the overall structure allows the plant holders to be supported on the single vertical central frame22. Alternatively, the rack20may include any number of vertical and/or horizontal elements, such as, for example, outer frame elements, to provide the rack20with sufficient support and strength. The gutters24are stacked and spaced vertically on top of one another on the central frame22, with each gutter24attached to the central frame22at approximately the longitudinal center of the gutter24to optimize balance. The rack20may include mounts31that hold the gutters24. Casters23may be disposed at the bottom of the central frame22to allow the rack20to move along a floor of the enclosure10. The rack20may be made of, for example, aluminum, plastic or other types of rigid material. In exemplary embodiments, the rack20is made using any suitable construction technique, such as, for example, welding or 3D printing.

FIG.8is a perspective view of the system1according to an exemplary embodiment of the present invention, showing the racks20moving on the conveyor system30through stationary scaffolding15that may hold, for example, a lighting system (including light fixtures) and irrigation system components. In the exemplary embodiment shown inFIG.8, the night-time cycle portion and the day-time cycle portion of the enclosure10are arranged in-line with one another so that each rack20follows a loop with long sections that are half in the night-cycle portion and half in the day-cycle portion. However, as shown inFIG.1, it should be appreciated that the night-time cycle portion and the day-time cycle portion of the enclosure10may be arranged side-by-side so that each rack20follows a loop with a long section that is entirely in the day-time cycle portion and another long section that is entirely in the night-time cycle portion. It should also be appreciated that the racks20may follow any other path within the enclosure10that allows for differences in lighting throughout a selected time period, with one or more sections of varying lengths.

As shown more clearly inFIG.9, the racks20are able to move freely between the scaffolding15and into alignment with the stationary irrigation stations70due to the cantilever structure of the scaffolding15. In this regard, as shown inFIG.10, the scaffolding15may include cross-pieces17that extend in the travel direction of the racks20, and the cross-pieces17may have openings through which sets of light fixtures18may extend in a horizontal direction (and transverse to the travel direction of the racks20). This allows the light fixtures18to be held in place in a cantilevered arrangement to allow the racks20to pass through the scaffolding without interference. For example, as shown inFIG.9, the central frame22of each rack20may pass between cantilevered light fixtures18that extend in opposite directions from both sides of each scaffolding15. In exemplary embodiments, the scaffolding15may hold a plurality of sets of light fixtures18, with each set arranged at a specific height above a corresponding one of the gutters24of the rack20. This arrangement allows all of the plants within each gutter24in each rack20to be exposed to an appropriate amount of light as the racks20traverse through the system1. The light fixtures18are held stationary on the scaffolding1as the racks20pass through each scaffolding15on the conveyor system40. This overall configuration is advantageous in that lighting components do not need to be moved around the enclosure10to simulate day-night cycles, which might otherwise require excessive wiring and cause accidents and/or result in damage to the lighting components or other components of the system1. Another advantage of this configuration is that less control of the lighting components is required, since lighting components can simply be omitted from the night-time cycle portion of the enclosure10rather than needing to turn off or dim the lighting to simulate night time. Alternatively, or in additionally, lighting components may be provided in the night-time cycle portion of the enclosure10that provide less light compared to lighting components provided in the day-time cycle portion of the enclosure10.

In exemplary embodiments, the light fixtures18may include light sources, such as, for example, incandescent, fluorescent, halogen, LED (light emitting diode), laser, or HID (high-intensity discharge) light sources, to name a few. The lighting system may include intensity controls and drivers so that the intensity of the light can be adjusted for different plant types and/or different parts of the growth cycle.

FIG.3is a more detailed view of the bottom of a rack20according to an exemplary embodiment of the present invention. The rack20includes a guide bar21to which is attached vertically oriented rollers26, in turn to which are attached the casters23. The guide bar21is fixed to the central frame22and extends generally parallel to the gutters24. As explained in more detail below, the casters23and rollers26are spaced sufficiently apart from one another to allow the casters23and rollers26to traverse along the conveyor system40, while the guide bar21provides the rack20with sufficient rigidity so that the rack20remains stable during movements within the conveyor system40. In exemplary embodiments, the rack20may include bumpers27disposed on the guide bar21and/or at any other portion of the rack20to avoid damage to the rack20in case of contact with other racks20or any other object that might be in the path of the rack20.

FIG.4is a more detailed view of the top of a rack20according to an exemplary embodiment of the present invention. A top mount assembly29is disposed at the top extremity of the rack20, the purpose of which is to attach to the conveyor system40, which may be an overhead conveyor. In this regard, the top mount assembly29may include a clamp, bracket or other structural component configured for attachment to the conveyor system40. In exemplary embodiments, the top mount assembly29includes a swivel so that the rack20remains in the same orientation around turns.

As shown most clearly inFIGS.3and4, each gutter24includes a series of plant holders25into which one or more plants and corresponding amounts of growing medium may be inserted. Although each gutter24is shown with sixteen plant holders25, it should be appreciated that each gutter24may include any number of plant holders25. Also, although the plant holders25are shown as being arranged in a single row, each gutter24may include any number of rows of plant holders25, with any number of plant holders25in each row.

Further, although the gutters24are shown as generally rectangular components, it should be appreciated that the gutters24may have any other shape, and the plant holders25may be arranged along any surface of the gutters24. In exemplary embodiments, the plant holders25are openings formed in the gutters24, where such openings may be circular in shape to accommodate circular plant pots or have any other suitable shape. In exemplary embodiments, the plant(s) in each plant holder25may or may not be held in pots. For example, plant(s) may be held directly in each plant holder25without corresponding plant pots. Further, in exemplary embodiments, the racks20may carry the plants in such a manner that the plant roots are exposed to allow for use of aeroponic cultivation systems, in which case the plant holders25may be omitted.

In exemplary embodiments of the present invention, each gutter24includes a top fill opening30and a side drain opening32. As explained in more detail below, the top fill opening30allows an irrigation station70to fill each gutter24with irrigation fluid and the side drain opening32allows for the irrigation station70to drain the irrigation fluid. It should be appreciated that each gutter24may include one or more drains located at any other positions around the gutter24, such as, for example, on the bottom of the gutter24, or may not include any drains. In exemplary embodiments, irrigation fluid may be drained directly from the gutters24to the floor of the enclosure10through vertical supports.

Conveyor System

FIGS.5and6show a bottom portion of a conveyor system40according to an exemplary embodiment of the present invention. The bottom portion of the conveyor system40includes a guide assembly41made up of internal guide rails42A, external guide rails42B and tracks44that guide the rollers26of the racks20so that the racks20follow a predetermined path within the enclosure10. In this regard, the guide rails42A,42B generally guide the racks20along straight sections of the path, while the tracks44generally guide the racks20along curved sections of the path. For example, the tracks44may be located within the conveyor system40where the racks20are shifted to another section of the path, and in this regard may include one or more toggle switches45. As shown inFIG.6, each rack20may be conveyed so that each caster23of the rack20follows a respective one of the tracks44and the rack20is shifted into position to follow another section of the path while ensuring the racks20continue to face the same direction. In the exemplary embodiment shown inFIG.6, as indicated by the arrows, one caster23of the rack20(in this case, the right caster23) has been guided from an external guide rail42B to internal guide rail42A through a switcher45, while the other caster23(in this case, the left caster23) has been guided from an internal guide rail42A to an external guide rail42B. The rollers26are spaced so as to remain in contact with the guide rails42A,42B while the rack20is moved along the conveyor system40. In exemplary embodiments, the casters23may or may not directly contact the floor of the enclosure10while the rack20is moving through the conveyor system40. For example, the casters23may not contact the floor while the rack20is moving along the guide rails42A,42B, but make contact with the floor (or a bottom surface of the tracks44) when the rack20is moving through the tracks44. In this regard, the casters23provide added stability to the rack20while the rack20is being switched to the opposite direction.

FIG.7is a top view of the conveyor system40according to an exemplary embodiment of the present invention. The conveyor system40includes a conveyor47that moves the racks20throughout the enclosure10and along the guide assembly41. The conveyor47may be an overhead conveyor, such as, for example, a powered overhead conveyor, a synchronous powered overhead conveyor, an asynchronous powered overhead conveyor (such as, for example, a power and free conveyor), an open track overhead conveyor, or a closed track overhead conveyor, to name a few. In exemplary embodiments, it should be appreciated that the conveyor system40is not limited to an overhead conveyor, and other exemplary embodiments may involve conveyors that drive the racks20from the bottom, from the bottom and top, or from any other point on the racks20. Further, it should be appreciated that the conveyor system40is not limited to the extent that the racks20are moved individually, and in other exemplary embodiments the racks20may be linked together and conveyed as a single unit. In exemplary embodiments, the conveyor47may include components, such as, for example, one or more chains, one or more trolleys, one or more brackets, one or more drive units, one or more take-up units, and one or more electrical control units, to name a few. Suitable conveyors are available from, for example, Rapid Industries (Louisville, Kentucky, USA), Ultimation Industries, LLC (Roseville, Michigan, USA), Daifuku (Osaka, Japan), and Richards-Wilcox Conveyor (Aurora, Illinois, USA), to name a few.

Irrigation System

FIG.11is a partial view of an irrigation station, generally designated by reference number70, according to an exemplary embodiment of the present invention. Any number of irrigation stations70may be placed within the enclosure10, and in exemplary embodiments the number of irrigation stations70may be within the range of five to fifteen, or less or more than this range. The irrigation station70includes a support structure71that holds a plurality of irrigation sub-assemblies74. Each irrigation sub-assembly74includes a tank76, a piston assembly78, a stopper79, a spigot assembly80, and a drain tray82. An overflow pipe84is in fluid connection with each of the tanks76, and a bottom end of the overflow pipe84is in fluid connection with main drainpipe86. Each irrigation sub-assembly74is arranged at a corresponding height so that as a rack20is moved into position next to the irrigation station70, each gutter24of the rack20is aligned with a corresponding one of the irrigation sub-assemblies74. As explained in more detail below, this allows for filling of each gutter24in the rack20with irrigation fluid and subsequent draining of the irrigation fluid from each gutter24before the rack20is moved downstream.

FIG.12shows flow of the irrigation fluid during filling and draining of the gutters24by the irrigation station70. Irrigation fluid is introduced to the irrigation station70from a main irrigation fluid feed at the top irrigation sub-assembly74where it enters and begins to fill the corresponding tank76. When the filling process begins, the piston assembly78moves the stopper79into engagement with the side wall of the gutter24, thereby blocking the side drain opening32of the topmost gutter24. Irrigation fluid is then fed from the tank76of the topmost irrigation sub-assembly74to the corresponding spigot assembly80, which in turn feeds the irrigation fluid into the gutter24through the top fill opening30. The draining process may commence after a predetermined period of time during which the plants within the gutter24are adequately soaked. The soaking period may be any suitable period of time, such as, for example, 30 seconds, one minute, or two minutes, to name a few.

During the draining process, the piston assembly78moves the stopper79away from the side drain opening32, thereby allowing the irrigation fluid from the top gutter24to drain onto the drain tray82of the top gutter24. The drain tray82guides the drained irrigation fluid from the topmost gutter24into the tank76of the next irrigation sub-assembly74just below the topmost irrigation sub-assembly. The next irrigation sub-assembly74can then perform the same filling and draining process for the gutter24just below the topmost gutter24using the associated piston assembly78, stopper79and spigot assembly80. The irrigation process then continues downward until the bottom most gutter has been irrigated and drained, with any overflow irrigation fluid within the tank being drained into the overflow pipe84and into the main drainpipe86. The main drainpipe86may be connected to other irrigation stations70throughout the enclosure10so that the irrigation fluid from each irrigation station70can re-circulate to the main irrigation fluid feed. In this regard, the main drainpipe86may be connected to a main tank (not shown) that holds irrigation fluid to be supplied to the main irrigation fluid feed at the top of each irrigation station70.

It should be appreciated that irrigation station70is not limited to the description provided above, and in other exemplary embodiments, each tank76of each irrigation sub-assembly74may be supplied separately with irrigation fluid rather than each sub-assembly74relying on the irrigation fluid being drained from the gutter24just above it, in which case irrigation fluid may be drained directly from the gutters24into a main drain pipe, for example. In another exemplary embodiment, each sub-assembly74may not have a corresponding tank76but instead may have a supply-drain line through which irrigation fluid is delivered to the top of the corresponding gutter24and then through which the irrigation fluid is pumped out of the gutter24.

FIGS.13A and13Bshow a gutter1024according to another exemplary embodiment of the present invention. The gutter1024includes a bottom1027, sides1028and a top1029. The top1029includes a plurality of openings1030configured to hold potted or unpotted plants. The height of the gutter1024varies from a maximum at a proximal end of the gutter1024to a minimum height at a distal end. The gutter1024may include a pocket1025at the proximal end of the gutter1024, which as described in more detail below, assists in the irrigation process.

During the irrigation process, an irrigation feed point70made up of a spigot72fills the gutter1024with irrigation fluid and then once filled, sucks the fluid out of the gutter1024. In this regard, the spigot72is automatically controlled to move into position into the gutter1024for filling, and then the same spigot72or a separate suction line (not shown) may be used to remove the fluid. The spigot72may be placed in position over the pocket1034to allow for more efficient filling of the gutter1024while avoiding overspill.

FIG.14shows an irrigation system, generally designated by reference number1030, according to an exemplary embodiment that may be used with the gutter1024. The irrigation system1030includes tanks1032, which may be located within or above the enclosure10. During the irrigation process, irrigation fluid is pre-filled into the tanks1032. Filling of the tanks1032may begin at the left-most tank1032through a valve, such as, for example, a ball valve or solenoid, and overflow into the tanks1032to the right. Each tank1032may include a water level sensor that detects when each tank1032has been filled to the appropriate volume of water. A lifting mechanism1034, such as, for example, a pneumatic cylinder may then be controlled to lower the spigots72into the gutters. Flex hoses1036may be used to allow the spigots72to lift and lower relative to stationary plumbing. Ball valves1038from each tank1032may then open to allow irrigation fluid to flow from the tanks1032to the gutters1024. After the gutters1024are soaked for a predetermined amount of time, the ball valves1038close and self-priming pumps1040may be turned on to pump the irrigation fluid from the gutters1024. A “Y” PVC fitting may be used to ensure irrigation fluid naturally flows to the gutters1024and not the pump during the fill sequence. After pumping is complete, the lifting mechanism1034is controlled to lift the spigots72from the gutters1024. An ultrasonic or other type of level sensor may be mounted to the end of the spigots72to detect if the fill and drain sequences are successful. In exemplary embodiments, low- and high-level sensors can be added to the tanks1032to ensure proper operation and/or overflow piping may be used to ensure volume of fluid in each tank does not exceed a predetermined amount (where “predetermined amount” may refer to a desired amount of fluid sent to the gutter when the valve opens). Also, in exemplary embodiments, the lifting mechanism1034may use proximity sensors to ensure proper movement.

FIG.15shows an irrigation system, generally designated by reference number1130, according to another exemplary embodiment that may be used with the gutter1024. The irrigation system1130includes tanks1132. Filling of the tanks1132may begin at the top-most tank1132through a valve, such as, for example, a ball valve or solenoid, and overflow into the tanks1132below. Each tank1132may include a water level sensor that detects when each tank1132has been filled to the appropriate volume of fluid. During a fill sequence, three-way valves1138, which may be motorized ball valves, are actuated to allow flow of irrigation fluid from the tanks1138into the gutters1024. After soak time, the three-way valves1138are reversed to connect pumps1140to the gutters1024. Each pump1140is turned on to pull irrigation fluid from the respective gutter1024and into the tank1138just below the pump1140. A lifting mechanism1134, such as, for example, a pneumatic cylinder may be controlled to lower and raise the spigots72relative to the gutters. Flex hoses1136may be used to allow the spigots72to lift and lower relative to stationary plumbing. Lifting of the spigots72allows the racks to index without interference, while lowering allows the flood and drain sequence to begin. An ultrasonic or other type of level sensor may be mounted to the end of the spigots72to detect if the fill and drain sequences are successful. In exemplary embodiments, low- and high-fluid level sensors can be added to the tanks1132to ensure proper operation. Also, in exemplary embodiments, the lifting mechanism1134may use proximity sensors to ensure proper movement.

In exemplary embodiments, drip irrigation techniques may be used to deliver water directly to individual pots. Normally, pressurized lines and flow controlling emitters are used to balance the amount of water delivered to each plant. In moving plant systems, however, it is often difficult to pressurize irrigation systems. For these types of systems, using gravity to move water is more practical.

FIGS.16A and16Bshow a drip irrigation system, generally designated by reference number1230, according to an exemplary embodiment of the present invention. The system1230provides a mechanism for delivering substantially equal volumes of water to pots with limited head height. Specifically, the system1230includes sub-assemblies1240(only one sub-assembly is shown inFIGS.16A and16B), with each sub-assembly1240associated with a corresponding gutter1024. The sub-assembly1240includes a stationary spigot1242, a funnel1244, a plurality of reservoirs1246, and a plurality of tubes1248each connected to a corresponding one of the plurality of reservoirs1248. The spigot1242supplies a volume of irrigation fluid to the funnel1244. The total volume of delivered irrigation fluid is enough to irrigate a total number X of plants held by the gutter1024at one time. The funnel1244has X number of openings at a base of the funnel1244. When irrigation fluid is added the funnel1244, the funnel openings split the fluid into X small streams. In this regard, the funnel1244gets narrower near the openings, thereby allowing a small amount of fluid to create consistent head height over the openings. This results in X number of streams with similar flow rates. Each stream from the funnel1244is captured in a corresponding one of the plurality of reservoirs1246. Each tube1248is connected to a base of a corresponding one of the plurality of reservoirs1248and routed to a corresponding one of the individual pots, thereby delivering fluid to the pot. If the tubes1248were connected directly to the funnel, without the intermediary reservoirs1246, differences in tube resistance and elevations would result in uneven distribution of the water. The reservoirs1246act as a buffer allowing the pre-partitioned amount of water to flow to an individual pot at whatever rate allowed by the tubing1248. Overflow channels1245can be added to the reservoirs1248to detect if an individual tube1248gets clogged.

It should be appreciated that various sensors and control modules may be used in the irrigation systems according to exemplary embodiments of the present invention to carry out delivery of irrigation fluid to the plants within the enclosure10in a controlled manner. For example, sensors may be used to sense flow, level and other parameters associated with the irrigation fluid, as well as operating state of components of the irrigation system, and information obtained by the sensors may be used by control modules to operate the various components of the irrigation system according to exemplary embodiments of the present invention. Accordingly, in exemplary embodiments of the present invention, the irrigation system may be partially or fully automated.

Environmental Control System

In exemplary embodiments, the system1further includes an environmental control system configured to maintain the target profiles (including but not limited to air temperature, relative humidity, air velocity, air particulate count, and carbon dioxide concentration) within the enclosure10. For example, the environmental control system may control the air temperature and/or other parameters within the enclosure10to vary through a 24-hour period (or any other predetermined photoperiod) to simulate morning, day and evening temperatures that optimize growth of the crop.FIG.17shows components of the environmental control system, generally designated by reference number100, according to an exemplary embodiment of the present invention. The temperature control system100includes one or more HVAC units102and one or more air circulation units104disposed within the enclosure10. The former primarily provides controls to air temperature and relative humidity while the latter focuses on air velocity. The HVAC units102may be located on the ceiling of or within the enclosure10, with each HVAC unit102primarily located within a corresponding day/night half of the enclosure10. The air circulation units104, which may include circulation fans, may be disposed on the scaffolding15at points throughout the enclosure10to circulate the environmentally conditioned air generated by the HVAC units102. This separation of air flow and HVAC units allows for enhanced airflow through the enclosure10while minimizing environmental variance, which in turn allows for minimization of energy consumption, reduced equipment sizing, and less restriction on the height of the overall system1.

In exemplary embodiments, the environmental control system100varies the air temperature, relative humidity, and air velocity within the enclosure so that, as each rack travels around the enclosure100between the day and night halves, the rack20encounters a temperature, humidity and velocity variation profile that simulates day-night environmental conditions. Said environmental variation may occur over a 24-hour period or some other predetermined period of time. For example, as shown inFIG.18, each rack20may proceed through an environmental variation profile within the predetermined time period with a minimum temperature range of 8° C. to 10° C. at greater than 85% relative humidity and a maximum temperature range of 25° C. to 30° C. with relative humidity between 60-80%. It should be appreciated that the present invention is not limited to these temperature or relative humidity ranges, and in other exemplary embodiments, the environmental conditions may be higher or lower than these ranges. For example, the minimum temperature range may be lower than 8° C. to 10° C. and the maximum temperature range may be higher than 25° C. to 30° C. Further, in exemplary embodiments, the humidity during the day may be controlled to be in the range of 60% to 80% relative humidity and the humidity during the night may be controlled to be in the range of 75% to 95% relative humidity. In this regard, the HVAC units102may be controlled using feedback from sensors, such as, for example, air temperature sensors, humidity sensors, wind speed sensors and CO2sensors, to name a few, located at various points within the enclosure10, either installed in fixed locations and/or which are fixed to the racks20so that the sensors can measure the full plant environment as the racks20move through the enclosure10. In a more specific example, each rack20encounters the minimum temperature of the temperature variation profile within the night half of the enclosure10and encounters the maximum temperature within the day half of the enclosure. The temperature and other environmental parameters may be controlled to gradually change to appropriate day ranges as the rack20makes its way into and through the day half and gradually change to appropriate night ranges as the rack20makes its way out of the day half and into the night half. The environmental conditions may be selected based on a number of factors, such as, for example, the type of crop, desired time to harvest, and energy efficiency, to name a few.

FIG.19shows an environmental control system, generally designated by reference number2100, according to an exemplary embodiment of the present invention. The environmental control system2100includes plenum areas within the enclosure10that facilitate air circulation. The plenum areas may include plenum walls2110A,2110B that separate the plenum areas from other areas of the enclosure10. In this regard, the plenum walls2110A,2110B may be made of insulative material, such as, for example, plastic sheeting, fabric, metal panels or any other suitable material. Some or all of the plenum walls2110A,2110B may include slits or other openings to allow for conditioned air to circulate between the plenum areas and the other areas of the enclosure.

As described previously, the environmental control system2100includes one or more HVAC units2102A,2102B and one or more air circulation units2104A,2104B disposed within the enclosure10. The HVAC units2102A,2102B may be located at the upper portion of the enclosure10, such as, for example, on the ceiling, with each HVAC unit2102A,2102B primarily located within a corresponding day/night half of the enclosure10. The air circulation units2104A,2104B may be disposed on the scaffolding15at points throughout the enclosure10to circulate the environmentally conditioned air generated by the HVAC units2102A,2102B. As shown inFIG.19, the plenum walls2110A,2110B may be arranged so as to separate the enclosure10into the day/night portions. For example, the plenum wall2110A may be arranged closest to a side wall of the enclosure and another plenum wall2110B may be arranged closest to an opposite side wall of the enclosure, with two other plenum walls2110C,2110D arranged between the two side plenum walls2110A,2110B, thereby forming a day portion2120at one side of the enclosure10and a night portion2130on the opposite side of the enclosure10. One or more air flow baffles2114may be arranged throughout the system2100to direct air flow in appropriate directions.

FIG.20shows conditioning and circulation of air within the enclosure10resulting from operation of the environmental control system2100. In the day portion2120, the conditioned air (indicated by arrows A) is sent from the HVAC unit2102A down the plenum area and into the bottom of the area of the enclosure10in which the racks20are housed. This air then traverses through the racks20(and associated plants), gaining heat and humidity. The recirculated air (indicated by arrows B) is sent back to the HVAC unit2012A and also mixed with the conditioned air via the air circulation units2104A. The circulated air can then be re-conditioned and circulated once again through the racks20. In the night portion2130, the conditioned air (indicated by the arrows C) is sent from the HVAC unit2102B down and into the bottom of the side plenum in a duct which throws cold air into the racks sideways. Warmed up air (indicated by arrows D) is sucked from the top and the bottom of the racks within ducts and returned back to the HVAC unit2102B to be conditioned. In exemplary embodiments, the air circulation units2104A,2104B are used to create climate uniformity.

In exemplary embodiments, cooling capacity can be provided by systems that includes components, such as, for example, unit coolers, ducted systems with air handlers, direct expansion units, and combinations thereof, to name a few. In exemplary embodiments, air can also be delivered directly to individual plants through use of air tubes, such as air tubes mounted in the same orientation as the light fixtures18described earlier.

Pest Management System

In an exemplary embodiment, the vertical farm system1includes a pest management system, generally designated by reference number200. As shown inFIG.21, the pest management system200includes a card210coated with adhesive that holds insects that might fly or crawl onto the card210. In this regard, the card210may hold common crop pests, such as, for example, aphids, thrips, beetles and mites, to name a few. These pests are typically within the size range of 0.5 mm to 10 mm, and are often difficult to see and/or identify with the human eye. Within a period of time, for example within a period of one or more hours, days and months, thousands of insects may crawl or fly onto the card210. In the process, the card210is scanned using a conventional flatbed scanner220to thereby generate a corresponding gigapixel image212of the card210. The gigapixel image212is then fed to a pest recognition artificial intelligence model230configured to analyze and identify any pests from large scale images of such pests within the gigapixel image212of the card210.

In exemplary embodiments, each enclosure10within a farm made up of a plurality of enclosures10may include one or more cards210located at various sections of the enclosure10. The one or more cards210in each section may be scanned individually or more than one card may be scanned at once to generate a composite of card images. In exemplary embodiments, all cards from the same enclosure10are scanned at once to generate a gigapixel image. In exemplary embodiments, each image212may have a size of, for example, 5 GB or more.

FIG.22shows a process for generating a pest recognition artificial intelligence model230according to an exemplary embodiment of the present invention. In step S1101of the process, training data is collected and stored in a database. The training data may include data associated with features of specific pest-types and tags associated with those pest-types. For example, in the case of an aphid, the training data may include data associated with a unique shape of the aphid and a tag associated with the aphid identified based on the unique shape. A computer vision API (application programming interface), such as, for example, AWS Rekognition API, Microsoft Computer Vision or Google Cloud Vision API, to name a few, may be used to generate the training dataset.

In step1103of the process, a neural network may be trained using the training data from step S1101. In this regard, the training data may be fed into a neural network algorithm that applies appropriate weights to input data, or independent variables, to determine an appropriate dependent variable, with one or more dependent variables being determined and combined to determine a final result (e.g., identification of an image of a pest within an image dataset and categorization of the identified pest). In exemplary embodiments, the neural network algorithm may be implemented using deep learning frameworks, such as, for example, Tensorflow, Keras, PyTorch, MxNet, Chainer Caffe, Theano, Deeplearning4j, CNTK, and Torch, to name a few.

In step S1105, the trained neural network is tested for performance. For example, the trained neural network may be tested for precision, recall, F1 score, accuracy, Intersection over Union (IoU), Mean Absolute Error (MAE), to name a few.

In exemplary embodiments, the pest recognition model230may be a machine learning recognition model, such as, for example, a Support Vector Machines (SVM) model, Bag of Feature Model, or a Viola-Jones Model, to name a few. In the exemplary embodiments, the pest recognition model230may be a deep learning image recognition model, such as, for example, Faster RCNN (Region-based Convolutional Neural Network), Single Shot Detector (SSD) or You Only Look Once (YOLO), to name a few.

In exemplary embodiments, the pest recognition AI model may generate reports indicating presence or non-presence of pests within sections of an enclosure10. In this regard,FIG.23shows an example of a report generated by the pest management system200, including class of pests identified in an enclosure, number of detections, percentage of each pest out of the total of all pests detected, to name a few. Links may also be provided to view detections and/or scans.

In exemplary embodiments, the results of the pest recognition model230in locating and identifying pests on the card210may be checked manually by a person viewing the card210and visually spotting any pests. If the pest recognition model and/or the manual inspection results in identification of a pest, appropriate action may then be taken to eliminate the pest from the enclosure10.

In exemplary embodiments, pests may be detected that are not in the training set for inspection. In this regard, unsupervised and/or semi-supervised learning algorithms can be used to detect pests outside of the original training set. Large unlabeled datasets of historical data plus a small subset of labeled data may be used to bootstrap AI training. Suitable techniques that may be used in this regard include few-shot learning and anomaly detection, among others.

Harvesting System

As mentioned previously, the system includes a harvesting station500, and in exemplary embodiments the harvesting station500is fully automated using integrated handling and machine vision tooling affixed to robotic manipulators, single-axis servo positioners, conveyors, machine vision techniques and artificial intelligence. In this regard,FIG.24is a block diagram of a harvesting station500according to an exemplary embodiment of the present invention. The harvesting station500includes one or more harvesting robots552-1,552-2. . .552-noperatively connected to a server560and a programmable logic controller (PLC)556. Each of the harvesting robots552-1,552-2. . .552-nmay be operatively connected with one or more corresponding edge devices554-1,554-2. . .554-n, one or more corresponding ethernet-IPC bridges555-1,555-2. . .555-n, one or more corresponding frame synchronization modules564-1,564-2. . .564-n, one or more corresponding calibration modules566-1,566-2. . .566-n, one or more corresponding aggregator modules574-1,574-2. . .574-n, and one or more corresponding safety modules576-1,576-2. . .576-n, all of which may exist within the server560. The server may also contain a 3D module572, an inference module568, a training module570, memory561and PLC module562. The modules of the server560may be made up of software components, hardware components, or combinations of hardware and software components. Further, one or more modules may be combined and/or one or more modules may be separated into sub-modules. Although only one server560is shown inFIG.24, it should be appreciated that multiple servers may be provided, with multiple enclosures10(or “farms”) including one or more harvesting stations500associated with one or more servers of the multiple provided servers. Also, although only one PLC556is shown inFIG.24, it should be appreciated that the harvesting station500may include multiple PLCs, with each PLC associated with one or more corresponding harvesting robots552-1,552-2. . .552-n.

The harvesting robots552-1,552-2. . .552-ninclude corresponding camera units553-1,553-2. . .553-n. In exemplary embodiments, the camera units553-1,553-2. . .553-nmay be stereoscopic red-green-blue-depth (RGBD) cameras, such as, for example, an Intel® RealSense™ D405 camera (Intel Corporation, Santa Clara, California, USA). Other types of cameras may be used, such as, for example, plain stereo, structured light, or solid-state LiDAR, to name just a few.

As explained in more detail below, the harvesting station500operates to identify ripe fruit within a closed view of the crop environment and harvest the ripe fruit without causing damage to the plants or environment. In exemplary embodiments, the harvesting station500may also be configured to count the number of flowers in the enclosure10for appropriate control of the pollination system300, to be described in more detail below. The harvesting robots552-1,552-2. . .552-nare fixtured to stationary platforms so that as the racks20move along the conveyor system40, the harvesting robots552-1,552-2. . .552-nare able to access the crops and carry out the harvesting process. As shown inFIGS.25A and25B, in exemplary embodiments, the harvesting robots552-1,552-2. . .552-nare six-axis robots and may include multiple joints and an end effector555. The end effector555may be a gripper configured to grasp a stem and snip the stem to remove ripe fruit or the gripper may have a more claw-like configuration to directly grasp the fruit and pull the fruit from the stem. In this regard, the end effector555may include a grip portion that holds a stem, and a separate snipping portion that snips the stem while the stem is being held by the gripper portion. This allows the end effector555to then place the still-gripped harvested fruit onto a tray or other storage/packaging component. As also explained in more detail below, the camera units553-1,553-2. . .553-noperate to capture images of the crops and surrounding environment to assist in the harvesting process. In exemplary embodiments, the harvesting robots may be commercially available robots, such as, for example, Yaskawa Motoman (Yaskawa America, Inc., Miamisburg, Ohio, USA) or FANUC LR Mate (FANUC America Corporation, Rochester Hills, MI, USA).

As shown inFIG.25A, the harvesting robots552-1,552-2. . .552-nmay be held stationary on scaffolding590. The scaffolding590may include multiple levels with any number of harvesting robots552-1,552-2. . .552-nsupported at each level so that the harvesting robots552-1,552-2. . .552-ncan access the plants held on the racks20. In this regard, as each rack20enters the harvesting station area, the rack20may be held stationary to allow time for the harvesting robots552-1,552-2. . .552-nto harvest the fruit. Once harvested, the fruit may be placed by the harvesting robots552-1,552-2. . .552-nonto trays or other temporary storage components which can then be transported by a separate conveyance system to a packaging station.

The edge devices554-1,554-2. . .554-noperate to process image data captured by the camera units553-1,553-2. . .553-ninto data that can be used to carry out various processes at the server560. In this regard, the edge devices554-1,554-2. . .554-nmay be devices, such as, for example, NVIDIA® Jetson Nano™ (NVIDIA Corporation, Santa Clara, CA, USA), soc (system on a chip), sbc (single board computer), Raspberry Pi (Cambridge, England, UK), Intel® Edison (Intel Corporation, Santa Clara, California, USA) and Intel® NUC, to name a few. In exemplary embodiments, the edge devices554-1,554-2. . .554-nrun the camera drivers and send information from the camera to the server560through the ethernet-IPC bridges555-1,555-2. . .555-n. In this regard, the ethernet-IPC bridges555-1,555-2. . .555nmay include, for example, a ZeroMQ bridge, a RabbitMQ bridge, WebRTC Gateway, or a gRPC bridge, to name a few. The edge devices554-1,554-2. . .554-nmay be configured to output data into memory, which may be, for example, serialized messages or payloads sent vie inter-process communication (e.g., shared memory, memory-mapped files, file descriptors, pipes, Unix domain sockets, etc.), along with a timestamp. The data placed into memory may be image data contained within a message container, where the message has a binary serialization format, such as, for example, Cap′n Proto, Protobuf, FlatBuffers and JSON, to name a few.

The ethernet-IPC bridges555-1,555-2. . .555nwithin server560receives input from the edge devices554-1,554-2. . .554-nand carries out operations, such as those described in more detail below. In this regard, messages are sent from the bridges at the edge devices554-1,554-2. . .554-nand received at a corresponding ethernet-IPC bridge555-1,555-2. . .555nat the server560, where they are then placed in server memory561. Server memory561(commonly referred to as IPC) is a module that facilitates communication between all modules in server560. InFIG.24, all connections/arrows within modules in server560are made using server memory561as a pass-through interconnection between modules. In exemplary embodiments, image messages may be passed from the edge devices554-1,554-2. . .554-nto ethernet-IPC bridge555-1. . .555-nwithin server560at a rate of, for example, 30 times per second.

The PLC module562is configured to communicate with the PLC556to obtain the operating state of the harvesting robots552-1,552-2. . .552-nand also provides instructions to the harvesting robots to perform harvesting, pruning, and other operations. These instructions include, but are not limited to: locations for picks, trajectories for the harvesting robot to execute picks, validation of successful/unsuccessful execution of picks, locations for placement of picked berries/fruits, and validation of successful/unsuccessful placement of picked berries/fruits. In this regard, the PLC module562may determine operating states of the harvesting robots552-1,552-2. . .552-n, such as, for example, where the robots are located, whether the robots are idle, and whether the robots are in a picking mode, to name a few. The PLC module562may communicate with the PLC556using conventional industrial communication protocols. The PLC module562places the robot operating state data into the memory module561for use by the other modules on the server560. The robot operating state data may be in a serialized memory format that describes what a particular robot or collection of robots is doing at a point in time.

Exemplary pseudocode for implementation of the PLC module562is as follows:

class PlcModule:def __init__ (self, plc_address, robots, plc_api_version) :plc_connection = establish_connection_with_plc (plc_address,plc_api_version)for robot_number in robots:self.robot_controllers.append(RobotController(robot_number=robot_number,plc_connection) )def run (self) :while True: # Run foreverfor robot in self.robot controllers:# Get index of current rackrack_index = read_from_plc (self.plc_connection, rack_tag)current_state = robot.determine_operating_state ( )if current_state == RobotState.IDLE:pass # Do nothing while the robot is idle.elif current_state == RobotState.CALIBRATING:robot.publish_position_and_state_to_memory_module ( )elif current_state == RobotState.SCANNING:gutter_index = read_from_plc (self.plc_connection,gutter_tag)robot.publish_position_and_state_to_memory_module (rack_index, gutter_index)elif current_state == RobotState.WAITING_FOR_PICK_DATA:pick_data = read_pick_data_from_safety_module ( )robot.send_pick_instructions_to_plc (pick_data)elif current_state == RobotState.PICKING:robot.publish_position_and_state_to_memory_module ( )result =read_pick_success_from_backend_verification_module ( )robot.send_pick_success_to_plc (result)elif current_state == RobotState.PLACEMENT:robot.publish_position_and_state_to_memory_module ( )else:raise ValueError (“Invalid state”)

FIG.26is a flowchart showing a process carried out by the PLC module562according to an exemplary embodiment of the present invention. In step S2601of the process, for each robot, the PLC module562reads status of the robot and/or the farm. In step S2603, if the PLC module562determines that the robot is idle, the robot is skipped and the next robot is analyzed. In step S2607, if it is determined that the robot is scanning, additional gutter information is read from the PLC556, and then the PLC module562broadcasts relevant notification messages to various other modules using the memory module561. In step2611, if the PLC module562determines the robot is calibrating, picking and/or performing placement, relevant notification messages are broadcast to various other modules using the memory module561. In step S2615, if the PLC module562determines that the robot is waiting for pick data, the PLC module562reads the pick data from the safety module576-nwhen the data is ready, and then outputs the pick data to the PLC556for a robot index. If the PLC module562cannot determine status of the robot (e.g., invalid or unknown state), the PLC module562will return an error message.

The frame synchronization modules564-1. . .564-nare configured to read directly from the memory module561to obtain image messages and robot operating state data and synchronize the robot operating states with a captured image. In this regard, the frame synchronization modules564-1. . .564-nmay receive a notification each time a robot552-1,552-2. . .552-nhas initiated an image scan, indicating that an appropriate image must be found from the scan event that matches the robot operating state. Since the robots552-1,552-2. . .552-nare moving during the scan event, the captured images may be blurry, and thus in exemplary embodiments, the frame synchronization modules564-1. . .564-nmay downsample to capture separate image frames. For example, the downsampling may be one frame per second, or some other frame capture rate. When PLC module562receives a scan event and verifies that the robot is not moving, the frame synchronization modules564-1. . .564-nmay select a captured image frame and output a sync frame message into the memory module561that includes information on the captured image frame and the corresponding robot operating state data. Accordingly, the captured image frame is synched with the robot operating state at a particular point in time.

Exemplary pseudocode for implementation of the frame synchronization modules564-1. . .564-nis as follows:

class FrameSyncModule:def __init__(self) :self.image_buffer = { }self.event_queue = [ ]def run (self) :while True:# Receive image message from camera feedimage_data, time_stamp = camera_subscriber_socket.receive ( )# Store binary image data in buffer with timestampself.image_buffer [time_stamp] = image_dataif self.image_buffer.size ( ) > buffer_limit:self.image_buffer.drop_images_older_than_duration (10) # Dropimages older than 10 secondsrobot_event_msg = event_subscriber_socket.receive ( )if robot_event_msg is not None:# Add this event to the queueself.event_queue.append (robot_event msg)# Check if we can process the oldest event in the queueif length (self.event_queue) != 0:event_msg = self.event_queue.pop (0)# Fetch matching image based on timestampimage =self.image_buffer.retrieve_closest_image (event_msg.time_stamp)# Republish image with robot_event dataimage_republish_msg = create_message (vision_msgs.Image,image, event_msg.time_stamp)image_publisher_socket.send (image_republish_msg.to_bytes ( ) )

FIG.27is flowchart showing a process carried out by a frame synchronization module564-naccording to an exemplary embodiment of the present invention. In step S2701, the frame synchronization module564-nreceives image data and timestamp from memory module561and the image data is stored in an image buffer indexed by the timestamp. If the image buffer is too large, oldest data may be dropped from the buffer to meet a maximum buffer limit. In step S2703, the frame synchronization module564-ndetermines if new event data is available. If so, the frame synchronization module564-nwill then add the new event data to an event queue. Otherwise, the frame synchronization module564-nwill check if the oldest event in the queue can be processed. In exemplary embodiments, an event can be processed if the event timestamp is near to the timestamp of an image in the image buffer. If such a match is found, the frame synchronization module564-nwill pack the event data into a message along with the retrieved image, and then broadcast the message to various modules in the pipeline using memory module561. The processed event data may then be removed from the queue.

The calibration modules566-1. . .566-nare configured to use the synch frame messages generated by the frame synchronization modules564-1. . .564-nto perform an initial calibration or update an existing calibration of the robots552-1,552-2. . .552-nand camera units553-1,553-2. . .553-n. In this regard, the PLC556may be placed into a calibration mode which causes a robot552-1,552-2. . .552-nto progress through a plurality of movements while sending associated captured images to the server560. The calibration modules566-1. . .566-nmay then use this information to perform intrinsic and extrinsic calibration of the camera units553-1,553-2. . .553-n.

Exemplary pseudocode for implementation of the calibration modules566-1. . .566-nis as follows:

class CalibrationModule:def __init__(self) :calibration_settings = load_config(″/configs/calibration.yaml″)# CHARUCOBOARD SN003 (MEFA) (calib.io)self.charucoboard = aruco.CharucoBoard create(calibration_settings)# Aruco detection params (turned for closer detections)self.aruco_dict = aruco.Dictionary_get(aruco.DICT_5X5_1000)self.aruco_detector_params =aruco.DetectorParameters_create(calibration_settings)def run(self):while True: # Run forevermsg_rgbd = self.receive_latest_rgbd_frame_from_camera( )# Skip frames that have already been processed.if msg_rgbd.index in scanIdxs:continue# Add current index to processed setscanIdxs.add (msg_rgbd.index)gray_img = extract_image_from_msg(msg_rgbd)############### CHARUCOBOARD Processing ################ Get arucoboard markersarucoCorners, arucoIds = cv2.aruco.detectMarkers(gray_img, self.aruco_dict, self.aruco_detector_params)if arucoIds is None: # Skipping frame, since no aruco IDs weredetected.continue# Get highly precise charuco cornerscharucoretval, charucoCorners, charucoIds =aruco.interpolateCornersCharuco (arucoCorners, arucoIds, gray_img, self.charucoboard)######################### Process transform#########################transform_3d = extract_transform_from_msg(msg_rgbd)# Add these corners and ids to our calibration arraysif charucoretval >self.aruco_detector_params.required_num_corners:charucoCornersAll.append(charucoCorners)charucoIdsAll.append(charucoIds)list of 3d transforms.append(transform_3d)else: # Skipping frame, since not enough charuco corners weredetectedcontinue################## Perform calibration ###################if len(list_of_3d transforms) > 4:# Perform intrinsic calibrationself.camera_matrix, self.distortion_coefficients, rvecs,tvecs = (cv2.aruco.calibrateCameraCharuco (charucoCornersAll,charucoIdsAll,self.charucoboard,gray_img.shape,))# Perform extrinsic calibration using handeye packagecalibrator = handeye.HandEyeCalibrator(setup=″Moving″)for r, t, transform_3d in zip(rvecs, tvecs,list_of_3d_transforms):transform_2d = create_2d_transformation_matrix(r, t)calibrator.add_sample(transform_3d, transform_2d)if calibrator.get_num_samples( ) >=calibrator.min_samples_required:# Solve for hand-eye transformtool_T_cam = calibrator.solve(method=solver)rotation_rmse, translation_rmse = (calibrator.compute_reprojection_error(tool_T_cam))# Add the calibration to the configmapcalib_data = {″date″: datetime.datetime.now( ).strftime(″%d/%m/%Y%H:%M:%S″) ,″num_samples″: calibrator.get_num_samples( ),″rotation.rmse″: rotation_rmse,″translation.rmse″: translation_rmse,″matrix″: tool_T_cam,″intrinsic.matrix″: self.camer_matrix,″intrinsic.distortioncoeffs″:self.distortion_coefficients,}# Update the configmap from K8supdate_calibration_settings_for_robot(″/configs/calibration.yaml″, calib_data)

FIG.28is a flowchart showing a process carried out by the calibration module566-naccording to an exemplary embodiment of the present invention. After initial loading of calibration configuration settings (e.g., calibration target information, robot coordinate system information, etc.), the calibration module566-nreceives image data and timestamp from memory module561(step S1-2801). In this step, duplicate input frames may be ignored, and image data may be unpacked. In step S1-2803, the calibration module566-ndetects and refines calibration points on a calibration target. In step S1-2805, if a sufficient number of calibration points have been detected in the frame, the calibration module566-nadds the robot transform, calibration points, and IDs to a buffer. In step S1-2807, if a sufficient number of robot transforms, calibration points, and IDs have been gathered from multiple images, the calibration module566-nperforms intrinsic and extrinsic camera calibration. Step S1-2807includes sub-steps, including step S2-2809, in which the calibration module566-ncalculates intrinsic camera parameters, 3D translation, and 3D rotation lists using known parameters from the calibration target and from all aggregated calibration points and IDs. Step S2-2809includes sub-steps, including step S3-2811, in which the calibration module566-nstarts eye-in-hand camera calibration, which loops over all robot transforms and intrinsic-generated 3D translation and rotations. Step S2-2809includes sub-steps, including step S4-2813, in which the calibration module566-nadds robot transform and intrinsic-generated 3D translation and rotation to the calibration backend. The backend will reject mathematically degenerate samples. In step S4-2815, if sufficient number of samples are collected after removal of mathematically degenerate samples, the calibration module566-ncalculates extrinsic transform. In step S4-2817, the calibration module566-nsaves the intrinsic and extrinsic calibration in Kubernetes (or other container orchestration tool) as a configmap.

The inference module568is configured to use the captured 2D images and generate messages that includes inference data that are placed into the memory module561. In this regard, the inference module568uses the result of the training module, i.e., the trained model, to perform inference on the incoming real-time data. The inference module568may perform operations, such as, for example, object detection, masking, ripeness detection, bounding boxes and keypoint detection, to name a few. In exemplary embodiments, the inference module568may use an object detection model and a separate keypoint detection model. In exemplary embodiments, the inference module568may perform its operations using one or more neural networks, such as, for example, Mask R-CNN, YOLOACT, Keypoint R-CNN, GSNet, Detectron2 and PointRend, to name a few. In exemplary embodiments, the inference module568may use one or more accelerators for enhanced speed and efficiency. Suitable accelerators include, for example, graphics processing units (GPUs), tensor processing units (TPUs), and field programmable gate arrays (FPGAs), to name a few. The input to the inference module568may be the synched frame messages generated by the frame synchronization modules564-1. . .564-nand the output may be an inference output message that includes robot operating state data, the original input message, depth (as part of RGBD data), masks, object detection, ripeness detection, bounding boxes, keypoints, and other relevant information.

Exemplary pseudocode for the inference module568is as follows:

class InferenceNode:″″″This class takes in a stream of RGB-D images and outputs inferenceresults from the AI models.″″″def __init__(self,path_to_mask_model,path_to_keypoint_model,):self.mask_model = MaskRCNNPredictor(path_to_mask_model)self.keypoint_model = KeypointPredictor(path_to_keypoint_model)def get_ripeness_score(self, img, mask):img_hsv = cv2.convert_to_hsv(img)red_pixels = cv2.inRange(img_hsv, min_red, max_red)num_red_pixels = cv2.countNonZero(red_pixels)num_berry_pixels = cv2.countNonZero(mask)ripeness_score = int(num_red_pixels * 100 / num_berry_pixels)return ripeness_scoredef berry_width_and_height(self, mask):big_strawberry_contour = self.find_biggest_contour(mask)berry_axes = self.circle_contour(mask, big_strawberry_contour)return berry_axesdef combine_boxes_based_on_iou(self, boxes1, boxes2, iou_threshold=0.5):iou_matrix = compute_iou_matrix(boxes1, boxes2)iou_matrix[iou_matrix < iou_threshold] = 0return nonzero_elements(iou_matrix)def infer(self, image):mask_outputs = self.mask model.infer_on_images(image)keypoint_outputs = self.keypoint_model.infer_on_images(image)# Combine outputs of models based on intersection over union (IoU) ofthe bounding boxes.combined_outputs = self.combine_boxes_based_on_iou(mask_outputs,keypoint_outputs)combined_outputs[″berry_data″] = [self.berry_width_and_height(mask)for mask in combined_outputs[″masks″]]return combined_outputs# output contains bounding boxes, detection masks, keypoints, majorand minor axes of the berry, and the label of the detected object.def run(self):while True: # Loop foreverfor msg in self.receive_message( ):image = extract_image_from_msg(msg)models_output = self.infer(image)if models_output.empty( ):continuemodels_output.filter_out_boxes_near_edge_of_image( )scores = [self.get_ripeness_score(detection) for detection inmodels_output]detections_msg = create_message(vision_msgs.ObjectDetections,models_output, scores)publish_detections_socket.send(detections_msg)visualization_msg = create_message(vision_msgs.Image, image,models_output)publish_visualization_socket.send(visualization_msg.to_bytes( ) )

FIG.29Ais a flowchart showing a process carried out by the inference module568according to an exemplary embodiment of the present invention. After initial loading of inference models, the inference module568received image data from the memory module561(step S1-2901). In step S1-2903, the inference module568runs inference on the image data. Step S1-2903may include sub-steps including step S2-2905in which the inference module568runs a mask and bounding box detection model, step S2-2907in which the inference module568runs a keypoint and bounding box detection model, step S2-2909in which the inference module568combines model outputs using, e.g., bounding boxes IoU and the Hungarian Algorithm, and step S2-2911in which the inference module568calculates berry width, height, and ripeness. In these steps, detections near the edge of an image may be discarded. In step S2-2913, the inference module568packages detections into a message and in step S2-2915broadcasts the message to various modules in the pipeline using memory module561.

FIG.29Bis an image of a picking environment before running the inference module568according to an exemplary embodiment of the present invention.FIG.29Cshows the output of the inference module, including masks, bounding boxes, keypoints and ripeness scoring. In exemplary embodiments, ripeness may be determined by the inference module568based on color of the fruit and/or other parameters. Ripeness scores may be based on a scale of 0 to 1, with lower range scores corresponding to “unripe” fruit, mid range corresponding to “ripening” fruit, and higher range scores corresponding to “ripe” fruit. It should be appreciated that the scoring is not limited to this scale or ranges.

The training module570prepares one or more object recognition and keypoint detection models, which may use neural networks. This is preferably run separately, and not as part of the real-time system. The training module570may train the models using a publicly available dataset for strawberries and/or other parts of the plant, such as, for example, StrawDI and the “strawberry picking point localisation ripeness and weight estimation” dataset. The datasets may be in standard formats, such as, for example, COCO, KITTI, and Cityscapes, to name a few. Alternatively, the dataset may be a proprietary dataset generated using creation, curation and annotation.

The 3D module572is configured to transform the captured 2D images into 3D image information based on the inference output message generated by the inference module568. In this regard the 3D module572may perform operations, such as, for example, calculation of width and height of a strawberry (in mm or other suitable unit of measurement), calculation of location of stem with respect to the camera, prediction for the percent of occlusion for a specific image, and addition of parameters to a transform tree that may include, for example, a global world frame, relative position of a robot, relative position of a camera and relative position of a strawberry, to name a few. The input into the 3D module572is the full RGBD data from the cameras553-1. . .553-nand the 2D keypoints and the 2D masks from the inference module568. The 3D module572integrates all three of these components, fills in any holes and corrects for camera calibration. The 3D module572may generate a set of 3D points representing the location of a strawberry with respect to a camera that captured an image of the strawberry. The location of a strawberry with respect to the global world frame may then be determined based on the known position of the robot from the robot operating state data and location of the strawberry with respect to the camera as determined by the 3D module572.

Exemplary pseudocode for implementation of the 3D module572is as follows:

class CalyxDet3DNode:″″″This node extracts calyxs from a RGBD image using bounding boxes as input.Returns a list of 3d points in camera frame.″″″def extract_single_bbox(self, depth_img, model_outputs, i):calyxDepth = get_calyx_depth(model_outputs.labelMasks, depth_img, i)return append_depth_to_2d_points(depth=calyxDepth, points_2d=[model_outputs.keypoints[i] ,model_outputs.majorAxes[i, 0],model_outputs.majorAxes[i, 1],model_outputs.minorAxes[i, 0],model_outputs.minorAxes[i, 1]])def run(self):while True: # Loop foreverfor msg_dets in self.receive_message( ):depth_img, model_outputs = msg_dets.parse( )intrinsics = getCameraIntrinsics( )# For calyx location in 3dcalyx3DPoints, axesMajorLeft3D, axesMajorRight3D, axesMinorLeft3D, axesMinorRight3D = [self.extract_single_bbox(depth_img, model_outputs, box)for box in model_outputs.boundingBoxes]# Transform 2D points to camera frame 3Dpointcloud = deproject_2d_to_3d(intrinsics, calyx3DPoints)# Transform all points into 3Daxes_3d = [deproject_2d_to_3d(intrinsics, ax) for ax in(axesMajorLeft3D, axesMajorRight3D, axesMinorLeft3D, axesMinorRight3D)]width3D, height3D = compute_3d_distances(axes_3d)col_sorted_indices = pointcloud.sort(″Z″)occlusion_dict = intersecting_boxes(model_output.boundingBoxes,model_outputs.labelMasks, col_sorted_indices)pick_ind = [ ]occ_scores = [ ]for ind in col_sorted_indices:if occlusion_dict[ind][″area″] > self.max_occlusion: # occludedby berries in the frontcontinuepick_ind.append(ind)occ_scores.append(occlusion_dict[ind][″area″] * 100.0)# Only select the indices that are not occludedfilter([model_outputs, pointcloud, width3D, height3D], by=pick_ind)msg_pc = create_message(vision_msgs.Pointcloud, model_outputs,(height3D, width3D), pointcloud, occ_scores)publish_socket.send(msg_pc.to_bytes( ) )

FIG.30is a flowchart showing a process carried out by the 3D module572according to an exemplary embodiment of the present invention. In step S1-3001. the 3D module572receives imaged data from memory module561. For each image received, the 3D module572carried out steps S2-3003to S2-3019. In step S2-3003, the 3D module572parses the depth image and model outputs and in step S2-3005, the 3D module572retrieves the camera intrinsics. In step S2-3007, the 3D module572extracts 2D points for all keypoints in each berry detection and appends a dimension to the tensor. In step S2-3009, the 3D module572converts the 2D points into 3D points using camera intrinsics and a depth projection function. In step S2-3011, the 3D module572computes 3D distances to determine the width and height of the berry in the 3D space. In step S2-3013, the 3D module572sorts the 3D detections by the Z-axis (depth) to prepare for occlusion handling. In step S2-3015, the 3D module identifies and filters out points that are significantly occluded based on a predefined maximum occlusion threshold. In step S2-3017, the 3D module packages occlusion scores, 3D keypoints, and associated model outputs into a message and, in step S2-3019, broadcasts the message to various modules in the pipeline using memory module561.

The aggregator modules574-1. . .574-nare configured to aggregate 3D image data to generate a world map of strawberries within a world frame using a plurality of 3D images. In this regard, the frame synchronization modules564-1. . .564-n, the inference module568and the 3D module572may “fire” only once per image so that a world map of strawberries is not known without aggregation of those images. In this regard, the aggregator modules574-1. . .574-nmay generate a world map using a plurality of collected 3D images, for example, one to sixteen images, to generate a world map of strawberries within a world frame. After the world map is projected onto the world frame, the aggregator modules574-1. . .574-nmay remove duplicate images and determine an ideal approach angle for the end effector555. The ideal approach angle may be determined by determining a least occluded image for a specific strawberry from the plurality of collected 3D images of that strawberry and then calculating the ideal approach angle based on the determined least occluded image.

Exemplary pseudocode for the implementation of the aggregator modules574-1. . .574-nis as follows:

class PointcloudAggregatorNode:def run(self):while True: # Loop forever# Get new pointcloud datapointcloud_msg = self.sub_pointcloud_sock.receive( )# Update tf tree with new transformself.transform_tree.add_tf_message(pointcloud_msg.transforms)# Transform pointcloud to self.pc_framepc_user = get_pointcloud_in_user_frame(pointcloud_msg,self.transform_tree)# Discard points that are too far awayvalid_positions = pc_user[X, :] < self.max_valid_distance# Filter out points with low ripeness scorevalid_ripeness = pointcloud_msg.scores > self.min_ripeness_scorefiltered_msg_data = filter_message_data(pointcloud_msg, valid_positions, valid_ripeness)# New scan, reset pointcloud and scan posesif pointcloud_msg.scanUid != cached_scan_uid:reset_cache( )aggregated_msg_data.append(filtered_msg_data)if aggregated_msg_data.pointclouds.num_points( ) >self.min_cluster_size:# Cluster pointclouds using DBSCAN and estimate pick-pointsclusterer = DBSCAN( )# Cluster based on y, zclusterer.fit(aggregated_msg_data.pointcloud[(Y, Z), :])clusters = clusterer.results( )# For each clusterfor i, cluster_values in clusters:(clusters [i].ripeness_score,clusters [i].centroid,clusters [i].median_error,) = compute_cluster_metrics(aggregated_msg_data,cluster_values)# Sort the scans based on distance to the clusterscan_distances = calculate_scan_distance_to_cluster(aggregated_msg_data, cluster_centroids[i])aggregated_msg_data.sort(by=scan_distances)# Pick the least occluded scanleast_occluded_view =min(aggregated_msg_data.occlusion_scores)best_scan_index, width, height, robotsensor_T_berry = (extract_info_from_scan(aggregated_msg_data,least_occluded_view))# Get the full transformfull_transform = self.ransform tree.lookup_transform(self.robot_user_frame,self.robot_eoat_pick_frame,robotsensor_T_berry,)# Get the euler anglesxzywpr = convert_transform_to_euler_ angles(full_transform)if not best_scan_index:# No valid scans for this cluster. Use default pickanglexzywpr[3:] = default_pick_angle# Save the pick angles.clusters[i].pick_angle = xzywpr[3:]clusters[i].height = heightclusters[i].width = width# Discard bad clusters based on error metricclusters = clusters[(median_error <= self.error_thresh)& (ripeness_scores > self.min_ripeness_score)]# Sort clusters by X coordinate (pick order front to back)pick_order = np.argsort(cluster_centroids[0, :])msg_pickpoints = create_message(vision.PickLocations, clusters[pick_order])self.robot_pick_locations_pub_sock.send(msg_pickpoints.to_bytes( ) )

FIG.31is a flowchart showing a process carried out by the aggregator module574-naccording to an exemplary embodiment of the present invention. In step S3101, the aggregator module574-ncontinuously receives pointcloud data and, in step S3103, updates the transformation tree with new transforms. In step S3105, the aggregator module574-ntransforms the pointcloud to a specified frame and filters out points based on distance and ripeness criteria. In step S3107, the aggregator module574-nresets data cache for new scans and aggregates filtered data. Once sufficient data is collected, in step S3109, the aggregator module574-nperforms clustering using, for example, Density-Based Spatial Clustering of Applications with Noise (DBSCAN) or Hierarchical Density-Based Spatial Clustering of Applications with Noise (HDBSCAN). In step S3111, for each cluster, the aggregator module574-ncalculates metrics, selects the least occluded scan, and determines the best pick points including angle, height, and width. Clusters may be filtered based on error and ripeness, and sorted for picking order. In step S3113, the aggregator module574-npackages pick data into a message and broadcasts the message to various modules in the pipeline using memory module561.

The safety modules576-1. . .576-nare configured to determine whether a specific strawberry pick is within bounds. In this regard, the safety modules576-1. . .576-nmay determine whether a specific pick violates one or more rules based on the output of the aggregator modules574-1. . .574-n. The one or more rules may relate to, for example, predetermined area within which the pick should be limited, angle of approach is within a predetermined safety angle, and whether the pick would cause a robot to function outside of safety parameters, to name a few.

Exemplary pseudocode for implantation of the safety modules576-1. . .576-nis as follows:

class SafetyModule( ):′′′This class subscribes to pick position messages and filters them based onset safety zones / limits.′′′def __init__ (self):# Load safety zones from config fileself.safety_settings = safe_load(open(″/settings/safety.yaml″))def run(self):while True: # Run foreverfor msg in self.subscriber.receive_messages( ):cluster_centroids =extract_cluster_centroids_from_received_message (msg)# Filter pick locations based on safety zones, discardingclusters that are too far awayvalid_clusters = self.safety_settings[″rack_min_x″] <cluster_centroids[X] < self.safety_settings[ ″rack_max_x″]valid_clusters = self.safety_settings[″rack_min_y″] <cluster_centroids[Y] < self.safety_settings[″rack_max_y″]valid_clusters = self.safety_settings[″rack_min_z″] <cluster_centroids[Z] < self.safety_settings[″rack_max_z″]output_msg = create_filtered_output_message (msg,valid_clusters)self.publisher.send(output_msg)

FIG.32is a flowchart showing a process carried out by safety module576-naccording to an exemplary embodiment of the present invention. After initialization in which the safety module576-nis configured with predefined safety zones (per-robot), including allowable bounds along the X, Y, and Z axes, the safety module576-n, for each incoming message (step S3201) containing the 3-D pick locations from a set of scans, performs steps S3203-S3209. In step S3203, the safety module576-nextracts the cluster centroids from the received message. In step S3205, the safety module576-nfilters the pick locations based on safety zones, discarding clusters based on the allowable bounds along the X, Y, and Z axes. In step S3207, the safety module576-ngenerates a filtered output message and, in step S3209, broadcasts the message to various modules in the pipeline using memory module561.

Pollination System

In exemplary embodiments, the system1may include a pollination system that stores one or more beehives and releases a number of bees on a periodic basis, where the number of bees is determined based on the number of flowers within the enclosure10or any other factors related to bee pollination. In this regard,FIG.33is a block diagram of a pollination system, generally designated by reference number300, according to an exemplary embodiment of the present invention. The pollination system300includes a bee station310, a server330and a camera robot350. The bee station310, server330and camera robot350may be in communication through a network380, such as, for example, a wide area network or a local area network. Each enclosure10may include one or more bee stations310.

FIGS.34A and34Bshow simplified block diagrams of opposite sides of a bee station, generally designated by reference number310, according to an exemplary embodiment of the present invention. The bee station310includes a hive312held within a hive enclosure314. The hive enclosure314may be any commercially available bee box, such as, for example, NATUPOL™ (Koppert Biological Systems, Inc., Howell, MI, USA). A bee gate system, generally designated by reference number320, is attached to the hive enclosure314. The bee gate system320includes a bee exit gate assembly322and a bee entrance gate assembly330arranged side by side with one another. As explained in more detail below, a vision system including a camera340is disposed on top of the bee exit gate assembly322and bee entrance gate assembly330to track movement of bees in and out of the hive so that the number of bees in the enclosure10can be controlled using the bee exit gate assembly322. The bee exit gate assembly322and the bee entrance gate assembly330share a top wall350, which may be made of a transparent material, such as, for example, plexiglass or transparent acrylic. A lighting system352is disposed below the bee gate system320to backlight the bees within the bee gate system320to allow the camera340to view the bees. The lighting system352may be, for example, an LED strip. The bee exit gate assembly322and the bee entrance gate assembly330also include a shared bottom wall351that may be made of a translucent material, such as, for example, frosted glass or translucent acrylic.

The bee exit gate assembly322includes a proximal portion324, a middle portion326and a distal portion328. The proximal, middle and distal portions324,326,328are divided by first and second gates323,325. As explained in more detail below, the gates323,325are controlled to allow only a predetermined number of bees out of the hive at one time depending on pollination requirements. In this regard, the first gate323may open first to allow some bees to enter the middle portion326from the proximal portion324, followed by closure of the first gate323and subsequent opening of the second gate325to allow bees from the middle portion326to enter the enclosure10through the distal portion328.

The bee entrance gate assembly330includes a trap door332that allows bees to enter the hive but does not allow any bees to exit the hive. In exemplary embodiments, the trap door332may be separately provided as part of the hive enclosure314or may be an integrated as part of the bee gate system320.

FIG.35is a top cross-sectional view of the bee gate system320according to an exemplary embodiment of the present invention. Slots354are formed through the portion of top wall350over the bee exit gate assembly322that accommodate movement of the first and second gates323,325between open and closed configurations. A separate slot (not shown) may be provided for the trap door332in the case the trap door332is provided as part of the bee gate system320(otherwise a separate slot may not be needed if the trap door332is provided separately with the hive enclosure314). The bee exit gate assembly322and the bee entrance gate assembly330are separated by a wall416. A platform402is disposed adjacent to the bee exit gate assembly322and bee entrance gate assembly330to support other components of the bee gate system320, such as, for example, components for controlling operation of the first and second gates323,325, which may include servo motors. Sensors may also be used to control operation of the first and second gates323,325, and may be located on the underside of the bee gate system320connected via bolts in holes409with a slot419for sensing.

FIG.36is a perspective view of the bee station310according to an exemplary embodiment of the present invention. As mentioned, the bee station310includes a hive enclosure314and a bee gate system320, both of which may be supported on a common base plate313. In addition to the components mentioned above, the bee gate system320further includes a top housing342, a bottom housing344and a middle housing346disposed between the top and bottom housings342,344. The top housing342encloses a computing unit, to be described in more detail below. The middle housing346encloses and functions as a focal length spacer for the camera340. The bottom housing344encloses the lighting system352, among other components. The bee exit gate assembly322and bee entrance gate assembly330are disposed between the bottom and middle housings344,346. The platform402supports components including, for example, a first servo motor348for the first gate323and a second servo motor349for the second gate325.

FIG.37is an exploded view of the bee gate system320according to an exemplary embodiment of the present invention. Disposed within the top housing342is a computing unit, which may include, for example, a printed circuit board364and a single board computer360. The single board computer360may be, for example, a Raspberry Pi (Cambridge, England, UK), a BeagleBoard (Michigan, USA) or a Nano Pi (Guangzhou, China), to name a few. The computing unit may also include a Power over Ethernet (POE) connection362, such as, for example, a Raspberry Pi PoE HAT.

As also shown inFIG.37, a spacer366is provided to separate the first and second gates323,325on an axle on the platform402, and sensors368A and368B corresponding to the first gate323and the second gate325, respectively, are provided below the platform402.

The bee gate system320is controlled by the single board computer360, which may receive power and data through the PoE connection and which is operatively connected to the camera340and the lighting system352. The printed circuit board364is operatively connected to the single board computer360, the two sensors368A,368B and the two motors348,349.

FIG.38is a block diagram showing various computer modules of a bee box computing unit, generally designated by reference number370, according to an exemplary embodiment of the present invention. The computing unit includes a bee detection module372, a bee exit gate middle portion bee count estimator module374, a bee tracker module376, a bee counter module378, an exit gate control module382, and a command logic module380.

The bee detection module372uses image data from the camera340to detect locations of bees within the various regions of the bee exit and bee entrance assemblies322,330. In this regard, the bee detection module372may return global bee location data associated with bee locations within the proximal, middle, and distal portions324,326,328of the bee exit assembly322and within the bee entrance assembly330. Each bee detected in the various regions may be provided with an (x,y) coordinate, where the x coordinate is relative to a horizontal axis and the y-coordinate is relative to a vertical axis.

Exemplary pseudocode for implementation of the bee detection module372is as follows:

class BeeDetectionModule( ):def __init__(self):# Get camera and associated buffersself.is_calibrated = Falseself.camera = PiCamera(resolution=(640, 480),sensor_mode=self.sensor_mode, framerate=self.framerate)self.raw_capture array = PiRGBArray(self.camera,size=self.camera.resolution)def run(self):′′′Runs forever′′′# Wait for camera to warm uptime.sleep(5)frame_num : int = 0recal_this_frame : bool = Falsefor frame in self.camera.capture_continuous(self.raw_capture_array,format=″bgr″, use_video_port=True):image_timestamp = nsec_since_boot( )# Get OpenCV imageimg = frame.array# Convert to grayscaleimg_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)# Make sure that the camera is calibration before running anydetection algorithmif not self.is_calibrated:self.calibrate_or_load_calibration_from_disk(img_gray)self.recal_this_frame = Falseelse:# Detect beesdets, region_fill = self.detectBees(img_gray)# Call Bee Tracker Module 376bypassDelta = self.getBypassBeeDelta(dets)# Send detection message (even if no bees detected)msg_dets = beebox_msgs.VisionDetections.new_message(nSecMsgSend=nsec_since_boot( ),nSecCam=image_timestamp,)# Send results from Bee Tracker Module 376msg_dets.beeDeltaBypass = bypassDelta# Send raw estimate of middle portion bee count,# to be refined by Middle Port Bee Count Estimator 374msg_dets.hive = [det for det in dets if det.region ==Region.HIVE.value]msg_dets.airlock = [det for det in dets if det.region ==Region.AIRLOCK.value]msg_dets.farm = [det for det in dets if det.region ==Region.FARM.value]msg_dets.bypass = [det for det in dets if det.region ==Region.BYPASS.value]# Calculate region fill ratiosmsg_dets.fillRatio.hive = region_fill[Region.HIVE.value-1] /float(self.mask_areas[Region.HIVE.value-1])msg_dets.fillRatio.airlock =region_fill[Region.AIRLOCK.value-1] /float(self.mask_areas[Region.AIRLOCK.value-1])msg_dets.fillRatio.farm = region_fill[Region.FARM.value-1] /float(self.mask_areas[ Region.FARM.value-1])msg_dets.fillRatio.bypass = region_fill[Region.BYPASS.value-1] / float(self.mask areas [Region.BYPASS.value-1])# Send messageself.pub_sock_det_regions_output.send(msg_dets)frame_num += 1# Dispose of previous frameself.raw_capture_array.truncate(0)# Subscribe to recalibration request(non-blocking)dat_recalibrate =self.sub_sock_calibrate_event.receive(non_blocking=True)if dat recalibrate is not None:# Delete calibration file on diskif os.path.isfile(self.calibration_image_path):os.remove(self.calibration_image_path)self.is_calibrated = False# Subscribe to count reset request (non-blocking)dat_reset = self.sub_sock_reset_event.receive(non_blocking=True)if dat_reset is not None:# Reset centroid trackerself.tracker = self.tracker =CentroidTracker(maxDistance=self.maxDistance,maxDisappeared=self.maxDisappeared)def detectBees(self, img_gray):dets = [ ]region_fill = [ ]# Perform Otsu thresholding on each region separatelyimg_morphology = np.zeros((img_gray.shape[0], img_gray.shape[1]) ,dtype=np.uint8)for (x, y, w, h) in self.mask_bbox_regions:# Clip_image to bounding boximg_clip = img_gray[y:y+h, x:x+w]# Threshold image using adaptive thresholding (one per region)thresh_otsu, img_region =cv2.threshold(img_clip, 0, 255, cv2.THRESH_BINARY_INV+cv2.THRESH_OTSU)# Copy result to morphology imageimg_morphology[y:y+h, x:x+w] = img_region# Bitwise AND our current adaptive thresholded image (bees are white)with our mask (white center)# This should turn all pixels outside of our mask black.img_masked = cv2.bitwise_and(img_morphology, img_morphology,mask=self.background_mask)# Clean up global adaptive threshold (remove small detection areascaused by dirt)img_separated = cv2.morphologyEx(img_masked, cv2.MORPH_OPEN, kernel3,iterations = 2)# Get number of pixels filled in each regionfor (x, y, w, h) in self.mask_bbox_regions:# Clip image to bounding boximg_clip = img_separated[y:y+h, x:x+w]# Record per-region fillregion_fill.append(cv2.countNonZero(img_clip))# Number and locate the blobs by getting connected componentsretval, labels, stats, centroids =cv2.connectedComponentsWithStats(img_separated)# Create bee detectionfor j in range(1, retval) : # Ignore first blob (background)# Get labeled regionr, c = int(centroids[j][1]), int(centroids[j][0])component_area = stats[j][cv2.CC_STAT_AREA]region = int(self.region_mask[r, c])# Remove bad blob sizes.if component_area < self.min_area or component_area >self.max_area:continue# Create detectiondet = beebox_msgs.VisionDet(uid=len (dets), region=region, x=c,y=r, area=int(component_area))dets.append( det )return dets, region_fill, img_separated

FIG.40is a flowchart showing a process carried out by the bee detection module372according to an exemplary embodiment of the present invention. The camera is initialized with specified resolution, sensor mode, and frame rate, and a raw capture array is set up to hold the camera's output. In step S1-4001, the bee detection module372continuously captures frames from the camera in an infinite loop and, in step S1-4003, converts the current frame to grayscale. In step S1-4005, the bee detection module372determines if the camera has been calibrated. If the camera has not been calibrated, then in step S1-4007the bee detection module372attempts calibration. If it is determined that the camera is calibrated (or after calibration in step S1-4007), the bee detection module372detects bees in the frame using thresholding and morphology operations (S1-4009). Step S1-4009includes sub-steps S2-4011to S2-4019. In step S2-4011, the bee detection module372calculates number of bees entering/exiting bee entrance assembly330by calling bee tracker module376. In step S2-4013, the bee detection module372calculates the number of bees and their positions within different regions (e.g., hive, airlock, farm, bypass) based on the processed image. In step S2-4015, the bee detection module372estimates region fill ratios based on the detected bees and predefined region masks. In step S2-4017, the bee detection module372generates a message with detection results including bee positions, counts, and region fill ratios and broadcasts the message to various modules in the pipeline. The bee detection module372may also check for recalibration and reset requests, responding by either recalibrating the camera or resetting the tracking system as needed.

The middle portion bee count estimator module374uses the bee location data to estimate current number of bees in the middle portion326of the bee exit assembly322. In this regard, the bee counter estimate module374may use an exponential filter to estimate current number of bees in the middle portion326. Exemplary pseudocode for implementation of the middle portion bee count estimate module374is as follows:def middlePortionBeeCountEstimatorModule (msg_vision_dets)→float:vis_dets_airlock=len(msg_vision_dets.airlock)#Modified version of Exponential moving average factor [0-1] for bee count in airlock.#Note: the sensor is more likely to undercount bees.#So we use a modified version of an exponential moving average to give more probablistic weight to higher readings.alpha=self. ALPHAif self.bees_in_airlock<=vis_dets_airlock:alpha=alpha*self. ALPHA MULTIPLIER #Give higher weight to this recent higher reading.self.bees_in_airlock=(1.-alpha)*self.bees_in_airlock+alpha*vis_dets_airlock#Publish the estimated bee count in airlockreturn self.bees_in_airlock

FIG.41is a flowchart showing a process carried out by the middle portion bee count estimate module374according to an exemplary embodiment of the present invention. In step S4101, the middle portion bee count estimate module374receives the current detection count of bees in the airlock from a message. In step S4103, the middle portion bee count estimate module374adjusts the bee count estimate using an Exponential Moving Average, modified to give more weight to higher recent readings. This adjustment accounts for the sensor's tendency to undercount by increasing the weight (alpha) when a higher count is observed. In step S4105, the middle portion bee count estimate module374returns updated estimated bee count for the airlock, usable in other modules.

The bee tracker module376tracks the number of bees leaving and entering the bee entrance assembly330. In this regard, bees may enter the bee entrance assembly330, but not necessarily enter the hive, and in some cases may leave the bee entrance assembly330without entering the hive at all. Thus, the bee tracker module376tracks bee trajectories within the bee entrance assembly330within a predetermined period of time to determine an increase or decrease in the number of bees within the enclosure10. The bee tracker module376may use a filtering technique to generate bee tracking data, where the filtering technique may include, for example, Kalman filtering, nearest neighbor, extended Kalman filtering, and unscented Kalman filtering, to name a few. The bee tracking data is then used by the bee tracker module376to generate bee count adjustment data to subtract or add to the bee count within the enclosure10.

Exemplary pseudocode for implementation of the bee tracker module376is as follows:#Called by Bee Detection Module372with bee detection data in same process def getBypassBeeDelta (self, dets)→int:. . .Given a list of bee detections,return the number of bees that have crossed the bypass region in this timestep.. . .dets_bypass=[det for det in dets if det.region==Region.BYPASS.value]centroids=np.array ([[det.x, det.y] for det in dets_bypass])#Update the trackerkeys, centroids, prevCentroids, tracks=self.tracker.update (centroids)#Check if any centroids have crossed the midway point of the bypass regionbypass_cx=self.mask_centers [Region. BYPASS. value-1, 0]#Make bypass_cx not an integer, so that it is impossible#for the math to land on the midway point#This is very important to prevent integrator runaway!bypass_cx=bypass_cx-0.5delta=0for x_new, x_old in zip (centroids [:, 0], prevCentroids [:, 0]):x_new_centered=x_new-bypass_cxx_old_centered=x_old-bypass_cxdx=np. sign (x_new_centered)-np. sign (x_old_centered)#dx will be in range [−2, 2]. 0 means same side or no movement.#Ignore same side movement or standing on the boundaryif dx==0:continue#Entering the hive is a negative dx movement (right to left)delta+=−1 if dx<0 else 1return delta

FIG.42is a flowchart showing a process carried out by bee tracker module376according to an exemplary embodiment of the present invention. In step S4201, the bee tracker module376filters the detections to only include those in the bypass region. In step S4201, the bee tracker converts the filtered detections into an array of centroids, representing the positions of detected bees. In step S4204, the bee tracker module376updates a tracker with the centroids, which manages tracking of individual bees across frames and returns updated centroid positions along with previous positions and tracking information. In step S4206, the bee tracker module376calculates a modified center point (‘bypass_cx’) for the bypass region to ensure it's a non-integer value. This adjustment is important for preventing exact matches with the center point, which could lead to errors in calculating movement directions. In step S4208, the bee tracker module376iterates through the current and previous centroid positions, calculating the change in position relative to the modified bypass center. In step S4210, for each bee, the bee tracker module376determines if the bee has crossed the midway point of the bypass region by checking the sign change in its position relative to ‘bypass_cx’. In step S4212, the bee tracker module376returns the net delta, representing the overall movement of bees across the bypass region in the current timestep.

The command logic module378generates control data for the exit gate control module382to cycle through opening of the first and second gates323,325. The control data may be based on a bee limit setting, current bee count data and a scheduled rest period. The scheduled rest period may occur upon commencement of a nighttime period, at which point the hive door may be closed, followed by a count reset and hive door opening at the beginning of the following daylight period. The command logic module378tracks the number of bees in the enclosure to generate bee count data based on the bee count adjustment data generated by the bee tracker module376, bee release data generated by the exit gate control module382(described below), and reset data.

Exemplary pseudocode for the command logic module378is as follows:

class CommandLogicController( ):# ConstsSTATE_FILE = ″/data/airlock state.beebox.AirlockStateWithCount″def __init__ (self) -> None:# Try to load beebox state, counts, limits from file.if os.path.isfile(self.STATE_FILE):with open (self.STATE_FILE, ″rb″) as f:self.airlock_state =beebox_msgs.AirlockStateWithCount.read(f)@propertydef hourLimit(self) -> bool:′′′Returns True if bee output should be disabled.hour = time.localtime( ).tm_hourreturn self.SCHEDULE_ENABLE and ( (hour < self.RELEASE_TIME) or (hour>= self.RECAPTURE_TIME) )@propertydef beeCount (self) -> float:′′′Returns normalized beecount.′′′return self.airlock state.beeCountFarm -self.airlock_state.beeDailyMindef updateBeeCountWithDelta(self, delta):′′′Update beecount′′′self.airlock_state.beeCountFarm += deltaself.airlock_state.beeDailyMin = min(self.airlock_state.beeCountFarm,self.airlock_state.beeDailyMin)if delta != 0:self.saveBeeCount( )@propertydef airlockIsFree (self) -> bool:′′′Returns True if airlock is empty.′′′return(self.bees_in_airlock <= self.AIRLOCK_FREE_THRESH) andself.airlockInPosition@propertydef airlockIsOccupied(self) -> bool:′′′Returns True if airlock is occupied.′′′return (self.bees_in_airlock > self.AIRLOCK_FREE_THRESH) andself.airlock InPosition@propertydef airlockInPosition (self) -> bool:′′′Returns True if airlock is in position.′′′return (self.last_msg_motor_state is not None) andself.last_msg_motor_state.isInPosition@propertydef airlockAcceptingCommands(self) -> bool:′′′Returns True if airlock is accepting commands.′′′return (self.last_msg_motor_state is not None) and (notself.last_msg_motor_state.isMoving)def saveBeeCount(self):with open(self.STATE_FILE, ′wb′) as f:self.airlock_state.write(f)def handleDailyCountReset(self):hour = time.localtime( ).tm_hourday = time.localtime( ).tm_mdayif hour == self.RESET_TIME and day != self.lastCountResetDate:self.airlock_state.beeCountFarm = 0self.airlock_state.beeDailyMin = 0self.lastCountResetDate = day# Save to diskself.saveBeeCount( )# Publish reset event to all subscribing nodesself.pub_sock_reset.send(msg)def run(self) -> None:′′′Runs the airlock controller.′′′# Close the gate on initmsg_airlock_command =beebox_msgs.AirlockStateOverride.new_message(nSecMsgSend=nsec_since_boot( ),state=AirlockState.LOCKED.value)self.pub_airlock_state.send(msg_airlock_command)framenum:int = 0while 1:bee_delta = 0# Get mode override (non-blocking)msg_override_state =self.sub_override_mode_sock.receive(non_blocking=True)if msg_override_state is not None:self.override_state_int = msg_override_state.state ifmsg_override_state.override else None# Get latest bee limit from API Node (non-blocking)msg_override_limit =self.sub_override_limit_sock.receive(non_blocking=True)if msg_override_limit is not None:self.airlock_state.beeLimit = msg_overrid_limit.data# Write to file (in case of restart)self.saveBeeCount( )# Get manual update of bee count from API Node (non-blocking)msg_override_count =self.sub_override_count_sock.receive(non_blocking=True)if msg_override_count is not None:self.airlock_state.beeCountFarm = msg_override_count.data -self.airlock_state.beeDailyMinself.airlock_state.beeDailyMin = 0self.saveBeeCount( )# Check if we need to reset our bee countself.handleDailyCountReset( )# BLOCK for new vision datamsg_vision_dets = self.sub_detection_sock.receive( )# Get motor state (non-blocking)msg_motor_state =self.sub_motor_state_sock.receive(non_blocking=True)if msg_motor_state is not None:self.last_msg_motor_state = msg_motor_state# Update bee count using bypass delta from vision stackbee_delta += msg_vision_dets.beeDeltaBypass# Calculate error (error = target − actual).bee_count_error = self.airlock_state.beeLimit − self.beeCountvis_dets_airlock = len(msg_vision_dets.airlock)# Vent bees? (Y/N) Defaults to ingress if at bee limit.should_vent = (bee_count_error > 0+epsilon)self.bees_in_airlock =middlePortionBeeCountEstimatorModule(msg_vision_dets)# Update new state considering priorities.new_state_int:int = Noneif self.airlockAcceptingCommands:# Gui overrideif self.override_state_int is not None:new_state_int = self.override_state_int# Hour limitelif self.hourLimit:new_state_int = AirlockState.LOCKED.value if(self.airlockIsFree) else AirlockState.OPEN_TO_FARM.value # do not lock abee inside the airlock# Ventingelif should_vent:airlock_bee_delta, new_state_int =self.cycle_airlock(msg_vision_dets)bee_delta += airlock_bee_delta# If at bee limit and there are no bees in the airlock, lockit.elif self.airlockIsFree:new_state_int = AirlockState.LOCKED.value# if no case matches, do not change state.else:new_state_int = None# update bee countself.updateBeeCountWithDelta(bee_delta)# Publish bee count to API Nodeif self.last_msg_motor_state is not None:# send the airlock state update messageself.pub_airlock_state.send(msg_airlock state_update)# Update the airlock state based onif new_state_int is not None:msg_airlock_command =beebox_msgs.AirlockStateOverride.new_message(nSecMsgSend = nsec_since_boot( ),state = new_state_int)self.pub_motor_command.send(msg_airlock_command)# Prevent sending double commands to airlock.self.last_msg_motor_state = Noneframenum += 1def cycle_airlock(self, msg_vision_dets) -> Tuple[int, int]:′′′Non-blocking function that will request an appropriate state changeof the airlock.If the state change, vents or ingests bees, it will return the beedelta.Returns bee_delta, new_state.new_state is an int or None if no state change is requested.′′′bee_delta = 0new_state_int = None# Only allow for state changes if the airlock is working properly.if self.airlockInPosition:# Change state to new state.transaction_dir = (−1 if self.airlockIsFree else 1)# Figure out the next statenew_state_int = self.last msg_motor_state.commandedState +transaction_dirnew_state_int = max(new_state_int, 0)new_state_int = min(new_state_int,AirlockState.OPEN_TO_FARM.value)if (new_state_int == self.last msg_motor_state.commandedState):# This means that we are already in the target state.# Do not start a new transaction and exit immediately.return 0, Noneif (new_state_int == AirlockState.LOCKED.value):# Make a note of the time we locked the airlock.# Thus, we can add some additional time to the airlock locktime# for estimating the count.self.time_last_locked = time.time( )# The airlock was locked.We are about to release bees in adirection. Keep track of the delta.elif (new_state_int == AirlockState.OPEN_TO_FARM.value):if (self.time last locked is not None) and ( (time.time( ) −self.time_last_locked) < self.AIRLOCK_FILTER_TIME ):# Not enough time to estimate the count.return 0, None# We have been locked for a while. Release the bees!bee_delta = ceil(self.bees_in_airlock −self.AIRLOCK_FREE_THRESH)# It might be possible for us to accidentally ingest bees usingthe airlock.elif (new_state_int == AirlockState.OPEN_TO_HIVE.value):# Check that there is not a huge mass of bees in the hiveside.# Otherwise, we should wait to open the hive door.if msg_vision_dets.fillRatio.hive >self.HIVE_REGION_OVERCROWDED_RATIO:return 0, None# Allow the bees to enter the airlockbee_delta = −1 * ceil(self.bees_in_airlock −self.AIRLOCK_FREE_THRESH)return bee_delta, new_state_int

FIG.43is a flowchart showing a process carried out by the command logic module378according to an exemplary embodiment of the present invention. As part of an initialization procedure, the command logic module378attempts to load state file from memory, and uses defaults if state file does not exist. In step S4301, the command logic module378performed a receive mode override in which any manual override commands are checked that might have been sent to alter the airlock's operation mode. In step S4303, the command logic module378updates the system with new bee population limits received from external sources and saves these updates to memory. This ensures that the system maintains the most current operational parameters, even after a restart. In step S4305, the command logic module378performs a manual update of bee count, allowing for manual corrections or adjustments to the bee count, with saving of these changes to the state file. In step S4307, if a reset is performed on a daily schedule, the command logic module378resets the bee count to zero and writes this reset state to memory. In step S4309, the command logic module378waits for and then processes new bee detection data. In step S4311, the command logic module378updates the controller with the latest state of the airlock's motor. In step S4313, the command logic module378adjusts the internal bee count based on the net change (delta) of bees detected passing through bee entrance assembly330by bee tracker module376. In step S4315, the command logic module378applies a series of logical checks and balances to decide the airlock's next state, integrating manual overrides, environmental conditions (time-based restrictions), and operational needs (e.g., venting excess bees or locking the airlock). In step S4317, the command logic module378calls the middle portion bee count estimator module374for an exponential moving average for bee count in the airlock. In step S4319, the command logic module378communicates the updated bee count and any changes in the airlock state to external systems. In step S4321, the command logic module378issued commands to change the airlock state based on the determined need, with new states saved to memory to ensure that the system can recover the current operational state after any interruption.

The exit gate control module382operates the first and second gates323,325based on the control data generated by the command logic control module380. Upon release of bees from the exit gate assembly322, the exit gate control module382generates bee release data based on the number of bees released. The bee release data is then fed back to the command logic module378for adjustment of the bee count. Exemplary pseudocode for implementation of the exit gate control module382is as follows:

class GateControl( ):def __init__(self) -> None:# Initialize gate statecurrent_state = self.get_sensor_state( )self.motor_angles = self.get_control_command(current_state)self.target_state = current_statedef handle_messages(self) -> None:′′′Handle incoming messages.′′′current_state = self.get_sensor_state( )# Only process new commands if we are not in motion.if not self.isMoving:# Block for new commands (with a timeout)new_state = self.sub_command_sock.receive( )if new_state is not None:# New state requested. Send it to the motors.self.send_motor_state_command(new_state)# Update motor commands (speed control, de-sticking, disengage,etc.)self.update_motors(current_state)# Publish node state.self.pub_state_sock.send(msg_motor_state_update)def run(self) -> None:′′′Runs the airlock controller.′′′while 1:self.handle_messages( )@propertydef hasDivergedFromTarget(self) -> bool:′′′Returns True if the motor has diverged from the target state.Will return false in the case of a jammed motor.′′′return (not self.isJammed) and (not self.isMoving) and(self.get_sensor_state( ) != self.target_state)@propertydef isInPosition (self) -> bool:′′′Returns True if the airlock is in position.′′′return (not self.isMoving) and ( (self.get_sensor_state( ) ==self.target_state) or self.IGNORE_SENSORS)@propertydef isMoving (self) -> bool:return not math.isclose(self.timestamp_last_actuation, 0.0)def update_pwm_output(self) -> None:self.hive_servo.angle = self.motor_angles[0] if self.motors_engagedelse Noneself.farm_servo.angle = −1.* self.motor_angles[1] ifself.motors_engaged else Nonedef update_motors(self, current_state: AirlockState) -> bool:′′′Updates the motor state based on the current state.Returns a bool if the Airlock state has converged.′′′if not self.isMoving:# If we are stuck in the absence of a command, try to recover tolast commanded state.if (self.isJammed) or (self.hasDivergedFromTarget and notself.IGNORE_SENSORS):self.send_motor_state_command(self.target_state)return# target_theta is a vector of the target angles for both motors.target_theta = self.get_control_command(self.target_state_low_level)# Linear interpolation of motor angles from last state to new targetstate based on current time and max speed of motors.self.motor_angles = linear_interpolate(self.last_motor_angles,target_theta, self.timestamp_last_actuation, time.time( ))# State is converged to target state.Deenergize actuators.# if math.isclose(actuation_percentage, 1.0) and ( (current_state ==self.target_state) or self.DEBUG_IGNORE_SENSORS):if (math.isclose(actuation_percentage, 1.0) and (current_state ==self.target_state or self.IGNORE_SENSORS)):self.isJammed = Falseself.actuator_retry_times = 0self.target_state_low_level = self.target_stateself.timestamp_last_actuation = 0.self.motors_engaged = Falseself.update_pwm_output( )return True# If still in motion from any command.if actuation percentage < 1.0:self.update_pwm_output( )return False# If no longer in motion, but not yet at convergence to target, weneed to do some state cycling.if self.actuator_retry_times < self.ACTUATOR_MAX_RETRY*2:# Start a retry cycle.self.actuator_retry_times += 1# Every other retry cycle, swap from old and new target states.# We use the old state, since this is a safe state to be in# This is a pseudo-call to ″send motor state command″self.target_state_low_level = self.last_state if(self.actuator_retry_times % 2) else self.target_stateself.timestamp_last_actuation = time.time( )self.last_motor_angles = self.motor_anglesreturn False# JAMMED! No need to handle this further here.# We will update other nodes of the jammed state and await newcommands.self.isJammed = Trueself.timestamp_last_actuation = 0.self.motors_engaged = Falseself.update_pwm_output( )return Falsedef send_motor_state_command(self, target_state: AirlockState) ->None:′′′This command only expects to be called whenever the airlock is notin motion,as it will override the current command.Care needs to taken to only execute commands when the airlock isnot in motion.′′′# If we are already in the target state, no need to move.if (not self.isJammed) and (target_state == self.get_sensor_state( )):returnself.last_state = self.target_state_low_levelself.last_motor_angles = self.motor_anglesself.target_state = target_stateself.target_state_low_level = target_stateself.timestamp_last_actuation = time.time( )self.actuator_retry_times = 0self.motors_engaged = Truedef get_sensor_state (self) -> AirlockState:′′′Returns the current state of the airlock based on sensor readingsusing a lookup table.′′′r, c = self.hive_feedback.value, self.farm_feedback.valuestate = AirlockState( self.state_lookup[r, c] )return statedef get_control_command(self, desired_state : AirlockState) ->np.ndarray:′′′Returns the angular command to send to the actuators.′′′hive, farm = np.where(self.state_lookup == desired_state.value)# Create angular commands (open is always positive)farm_cmd = self.SERVO_CLOSED_DEG if (farm == 0) elseself.FARM_SERVO_OPEN_DEGhive_cmd = self.SERVO_CLOSED_DEG if (hive == 0) elseself.HIVE_SERVO_OPEN_DEGreturn np.asarray( [hive_cmd, farm_cmd] )

FIG.44is a flowchart showing a process carried out by the exit gate control module382according to an exemplary embodiment of the present invention. As part of an initialization procedure, the exit gate control module382acquires the initial gate state from sensors and determines motor angles needed to achieve or maintain this state. The target state is set to match the current state, establishing a baseline for operation. In step S4401, the exit gate control module382, at each iteration, retrieves the current gate state from sensors to understand the gate's real-time position. In step S4403, if the gate is not currently moving (‘isMoving’ property returns ‘False’), the exit gate control module382listens for new state commands from a subscribed socket. Upon receiving a command, the exit gate control module382issues a directive to adjust the motors accordingly (‘send_motor_state_command’ method), aiming to transition the gate to the requested state. In step S4407, the exit gate control module382, regardless of movement, updates motor commands to manage speed, address potential sticking issues, and disengage motors if necessary. In step S4409, the exit gate control module382, after handling the incoming messages and updating the motor state, creates a message with the current motor state and, in step S4411, broadcasts the message to various modules in the pipeline.

FIG.39is a flowchart showing operation of the exit gate assembly322according to an exemplary embodiment of the present invention. The process shown inFIG.39may be repeated on a periodic basis, such as, for example, every 5 seconds, or every 10 seconds, or every minute, or any other period. At step S01of the process, the first gate323is opened to allow bees to enter the first and second portions324,326of the bee exit assembly322. At step S03, the first gate323is closed, which may occur at a preset time after the first gate323is initially opened. At step S05, both the first and second gates323,325are kept closed for a period of time to allow the bee count to settle. This step provides adequate time for the bee tracker module376to track the number of bees entering and leaving the bee entrance assembly330and the middle portion bee count estimator module374to estimate current number of bees in the middle portion326. In step S07, the second gate325is opened, thereby releasing the bees from the middle portion326. In step S10, after the middle portion326is determined to be empty (step S09), the bee release data is sent to the bee counter module378for adjustment of the bee count within the enclosure10. In step S11, the second gate325is then closed.

In exemplary embodiments, the camera robot350is on a stationary platform and may include a vision system configured to identify and count the number of flowers in the crops as the racks20move past the robot350. In other exemplary embodiments, the flower count may be determined using a vision system integrated within the harvesting system500, for example, within the harvesting robots552-1,552-2. . .552-n. The vision system may be configured to recognize flowers in various stages of growth and provide fruit ripeness analytics. The vision system may implement machine vision image processing techniques, such as, for example, stitching/registration, filtering, thresholding, pixel counting, segmentation, edge detection, color analysis, blob detection and extraction, neural net/deep learning/machine learning processing pattern recognition including template matching, gauging/metrology, comparison against target values to determine a “pass or fail” or “go/no go” result, to name a few.

It should be appreciated that the bee station as described herein is not limited to use in an indoor vertical farm environment, and in other exemplary embodiments, the inventive bee station may be used in other agriculture environments, such as, for example, outdoor farming, indoor farming, conventional farming, and greenhouse farming, to name a few. For the purposes of the present disclosure, for a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.

The term “data processing apparatus” encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

As used in this specification, an “engine,” or “software engine,” refers to a software implemented input/output system that provides an output that is different from the input. An engine can be an encoded block of functionality, such as a library, a platform, a software development kit (“SDK”), or an object. Each engine can be implemented on any appropriate type of computing device, e.g., servers, mobile phones, tablet computers, notebook computers, music players, e-book readers, laptop or desktop computers, PDAs, smart phones, or other stationary or portable devices, that includes one or more processors and computer readable media. Additionally, two or more of the engines may be implemented on the same computing device, or on different computing devices.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit).

Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) monitor, an LCD (liquid crystal display) monitor, or an OLED display, for displaying information to the user, as well as input devices for providing input to the computer, e.g., a keyboard, a mouse, or a presence sensitive display or other surface. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending resources to and receiving resources from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (“LAN”) and a wide area network (“WAN”), e.g., the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a sub combination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

FIG.45is a block diagram illustrating an exemplary machine learning platform for implementing various aspects of this invention, according to some exemplary embodiments of the present invention.

System1500may include data input engine1510that can further include data retrieval engine1504and data transform engine1506. Data retrieval engine1504may be configured to access, interpret, request, or receive data, which may be adjusted, reformatted, or changed (e.g., to be interpretable by other engines, such as data input engine1510). For example, data retrieval engine1504may request data from a remote source using an API. Data input engine1510may be configured to access, interpret, request, format, re-format, or receive input data from data source(s)1502. For example, data input engine1510may be configured to use data transform engine1506to execute a re-configuration or other change to data, such as a data dimension reduction. Data source(s)1502may exist at one or more memories and/or data storages. In some embodiments, data source(s)1502may be associated with a single entity (e.g., organization) or with multiple entities. Data source(s)1502may include one or more of training data1502a(e.g., input data to feed a machine learning model as part of one or more training processes), validation data1502b(e.g., data against which at least one processor may compare model output with, such as to determine model output quality), and/or reference data1502c. In some embodiments, data input engine1510can be implemented using at least one computing device. For example, data from data sources1502can be obtained through one or more I/O devices and/or network interfaces. Further, the data may be stored (e.g., during execution of one or more operations) in a suitable storage or system memory. Data input engine1510may also be configured to interact with data storage, which may be implemented on a computing device that stores data in storage or system memory. System1500may include featurization engine1520. Featurization engine1520may include feature annotating and labeling engine1512(e.g., configured to annotate or label features from a model or data, which may be extracted by feature extraction engine1514), feature extraction engine1514(e.g., configured to extract one or more features from a model or data), and/or feature scaling and selection engine1516. Feature scaling and selection engine1516may be configured to determine, select, limit, constrain, concatenate, or define features (e.g., AI features) for use with AI models. System1500may also include machine learning (ML) modeling engine1530, which may be configured to execute one or more operations on a machine learning model (e.g., model training, model re-configuration, model validation, model testing), such as those described in the processes described herein. For example, ML modeling engine1530may execute an operation to train a machine learning model, such as adding, removing, or modifying a model parameter. Training of a machine learning model may be supervised, semi-supervised, or unsupervised. In some embodiments, training of a machine learning model may include multiple epochs, or passes of data (e.g., training data1502a) through a machine learning model process (e.g., a training process). In some embodiments, different epochs may have different degrees of supervision (e.g., supervised, semi-supervised, or unsupervised). Data into a model to train the model may include input data (e.g., as described above) and/or data previously output from a model (e.g., forming recursive learning feedback). A model parameter may include one or more of a seed value, a model node, a model layer, an algorithm, a function, a model connection (e.g., between other model parameters or between models), a model constraint, or any other digital component influencing the output of a model. A model connection may include or represent a relationship between model parameters and/or models, which may be dependent or interdependent, hierarchical, and/or static or dynamic. The combination and configuration of the model parameters and relationships between model parameters discussed herein are cognitively infeasible for the human mind to maintain or use. Without limiting the disclosed embodiments in any way, a machine learning model may include millions, trillions, or even billions of model parameters. ML modeling engine1530may include model selector engine1532(e.g., configured to select a model from among a plurality of models, such as based on input data), parameter selector engine1534(e.g., configured to add, remove, and/or change one or more parameters of a model), and/or model generation engine1536(e.g., configured to generate one or more machine learning models, such as according to model input data, model output data, comparison data, and/or validation data). Similar to data input engine1510, featurization engine1520can be implemented on a computing device. In some embodiments, model selector engine1532may be configured to receive input and/or transmit output to ML algorithms database1590. Similarly, featurization engine1520can utilize storage or system memory for storing data and can utilize one or more I/O devices or network interfaces for transmitting or receiving data. ML algorithms database1590(or other data storage) may store one or more machine learning models, any of which may be fully trained, partially trained, or untrained. A machine learning model may be or include, without limitation, one or more of (e.g., such as in the case of a metamodel) a statistical model, an algorithm, a neural network (NN), a convolutional neural network (CNN), a generative neural network (GNN), a Word2Vec model, a bag of words model, a term frequency-inverse document frequency (tf-idf) model, a Generative Pre-trained Transformer (GPT) model (or other autoregressive model), a Proximal Policy Optimization (PPO) model, a nearest neighbor model (e.g., k nearest neighbor model), a linear regression model, a k-means clustering model, a Q-Learning model, a Temporal Difference (TD) model, a Deep Adversarial Network model, or any other type of model described further herein.

System1500can further include predictive output generation engine1540, output validation engine1550(e.g., configured to apply validation data to machine learning model output), feedback engine1570(e.g., configured to apply feedback from a user and/or machine to a model), and model refinement engine1560(e.g., configured to update or re-configure a model). In some embodiments, feedback engine1570may receive input and/or transmit output (e.g., output from a trained, partially trained, or untrained model) to outcome metrics database1580. Outcome metrics database1580may be configured to store output from one or more models, and may also be configured to associate output with one or more models. In some embodiments, outcome metrics database1580, or other device (e.g., model refinement engine1560or feedback engine1570) may be configured to correlate output, detect trends in output data, and/or infer a change to input or model parameters to cause a particular model output or type of model output. In some embodiments, model refinement engine1560may receive output from predictive output generation engine1540or output validation engine1550. In some embodiments, model refinement engine1560may transmit the received output to featurization engine1520or ML modeling engine1530in one or more iterative cycles.

Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain implementations, multitasking and parallel processing may be advantageous.

Having thus described the present invention in detail, it is to be appreciated and will be apparent to those skilled in the art that many physical changes, only a few of which are exemplified in the detailed description of the invention, could be made without altering the inventive concepts and principles embodied therein. It is also to be appreciated that numerous embodiments incorporating only part of the preferred embodiment are possible which do not alter, with respect to those parts, the inventive concepts and principles embodied therein. The present embodiment and optional configurations are therefore to be considered in all respects as exemplary and/or illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all alternate embodiments and changes to this embodiment which come within the meaning and range of equivalency of said claims are therefore to be embraced therein.