Source: http://www.patentsencyclopedia.com/app/20140098237
Timestamp: 2019-04-20 05:00:33+00:00

Document:
Techniques are provided to implement line based processing of thermal images and a flexible memory system. In one example, individual lines of a thermal image frame may be provided to an image processing pipeline. Image processing operations may be performed on the individual lines in stages of the image processing pipeline. A memory system may be used to buffer the individual lines in the pipeline stages. In another example, a memory system may be used to send and receive data between various components without relying on a single shared bus. Data transfers may be performed between different components and different memories of the memory system using a switch fabric to route data over different buses. In another example, a memory system may support data transfers using different clocks of various components, without requiring the components and the memory system to all be synchronized to the same clock source.
1. A device comprising: a plurality of components adapted to transfer thermal image data; a plurality of buses connected to the components; and a memory system connected to the components by the buses, the memory system comprising a plurality of memory buffers, each memory buffer comprising: a memory block comprising a single interface adapted to support a single read or write operation at a time, a plurality of ports, wherein each port is adapted to communicate with a corresponding one of the components over a corresponding one of the buses, and a switch fabric block adapted to selectively couple one of the ports with the memory block to permit transfer of at least a portion of the thermal image data between the corresponding one of the components and the memory block through the coupled port and the single interface.
2. The device of claim 1, wherein each memory buffer is adapted to store a different single line of a thermal image frame included in the thermal image data.
3. The device of claim 2, wherein each memory buffer is a pipeline register for a line based image processing pipeline used to process the thermal image frame.
4. The device of claim 1, wherein each port is adapted to receive a clock signal from the corresponding one of the components over the corresponding one of the buses, wherein the memory block is adapted to be synchronized by the clock signal received by the coupled port.
5. The device of claim 4, further comprising a clock switch adapted to switch between the clock signals to synchronize the memory block in a glitchless manner.
6. The device of claim 1, wherein the switch fabric comprises a multiplexer adapted to select the ports based on control signals received by the memory buffer.
7. The device of claim 6, wherein the memory system further comprises a memory controller adapted to provide the control signals to select the ports in accordance with a predetermined instruction sequence to transfer the thermal image data.
8. The device of claim 1, wherein: a first subset of the components is connected to a first subset of the memory buffers by the buses, but is not connected to a second subset of the memory buffers by the buses; and a second subset of the components is connected to the second subset of the memory buffers by the buses, but is not connected to the first subset of the memory buffers by the buses.
9. The device of claim 1, wherein: a first one of the components adapted to communicate with a first one of the memory buffers and a second one of the memory buffers; a second one of the components is adapted to communicate with the first memory buffer but not the second memory buffer; and the first component is adapted to read the thermal image data from the second memory buffer and write the thermal image data to the first memory buffer to share the thermal image data with the second component.
10. The device of claim 1, wherein: a first one of the components is an application specific integrated circuit (ASIC); a second one of the components is a general purpose processor; a third one of the components is a non-volatile memory block of the memory system; and the memory block is a static random access memory (SRAM) block.
11. A method of operating a memory system and a plurality of components connected to the memory system by corresponding buses, the method comprising: operating a switch fabric to couple a first port of a memory buffer with a memory block of the memory buffer, wherein the memory block comprises a single interface adapted to support a single read or write operation at a time, wherein the memory buffer is part of the memory system; transferring first thermal image data between a first one of the components and the memory block over a first one of the buses, through the first port, and through the single interface; subsequently operating the switch fabric to couple a second port of the memory buffer with the memory block; and transferring second thermal image data between a second one of the components and the memory block over a second one of the buses, through the second port, and through the single interface.
12. The method of claim 11, further comprising storing in the memory buffer a single line of a thermal image frame included in the thermal image data.
13. The method of claim 12, wherein the memory buffer is a pipeline register for a line based image processing pipeline used to process the thermal image frame.
14. The method of claim 11, further comprising: receiving a first clock signal from the first component over the first bus and at the first port; receiving a second clock signal from the second component over the second bus and at the second port; synchronizing the memory block by the first clock signal while the first port is coupled with the memory block; and synchronizing the memory block by the second clock signal while the second port is coupled with the memory block.
15. The method of claim 14, further comprising switching between the first and second clock signals to synchronize the memory block in a glitchless manner.
16. The method of claim 11, further comprising receiving control signals at the memory buffer, wherein the operating steps are performed by a multiplexer of the switch fabric to select the first or second ports.
17. The method of claim 16, further comprising providing the control signals from a memory controller of the memory system to select the first or second ports in accordance with a predetermined instruction sequence to transfer the thermal image data.
18. The method of claim 11, wherein: the memory system comprises a plurality of memory buffers; a first subset of the components is connected to a first subset of the memory buffers by the buses, but is not connected to a second subset of the memory buffers by the buses; and a second subset of the components is connected to the second subset of the memory buffers by the buses, but is not connected to the first subset of the memory buffers by the buses.
19. The method of claim 11, wherein: the memory buffer is a first memory buffer; the first component is adapted to communicate with the first memory buffer and a second memory buffer; the second component is adapted to communicate with the first memory buffer but not the second memory buffer; and the method further comprises: reading the thermal image data from the second memory buffer using the first component, and writing the thermal image data to the first memory buffer using the first component to share the thermal image data with the second component.
20. The method of claim 11, wherein: the first component is an application specific integrated circuit (ASIC); the second component is a general purpose processor; a third one of the components is a non-volatile memory block of the memory system; and the memory block is a static random access memory (SRAM) block.
 This application is a continuation of International Patent Application No. PCT/US2012/41756 filed Jun. 8, 2012 which claims priority to U.S. Provisional Patent Application No. 61/646,750 filed May 14, 2012 and entitled "LINE BASED IMAGE PROCESSING" which are both hereby incorporated by reference in their entirety.
 International Patent Application No. PCT/US2012/41756 claims the benefit of U.S. Provisional Patent Application No. 61/646,732 filed May 14, 2012 and entitled "FLEXIBLE MEMORY SYSTEM" which is hereby incorporated by reference in its entirety.
 International Patent Application No. PCT/US2012/41756 claims the benefit of U.S. Provisional Patent Application No. 61/545,056 filed Oct. 7, 2011 and entitled "NON-UNIFORMITY CORRECTION TECHNIQUES FOR INFRARED IMAGING DEVICES" which is hereby incorporated by reference in its entirety.
 International Patent Application No. PCT/US2012/41756 claims the benefit of U.S. Provisional Patent Application No. 61/495,873 filed Jun. 10, 2011 and entitled "INFRARED CAMERA PACKAGING SYSTEMS AND METHODS" which is hereby incorporated by reference in its entirety.
 International Patent Application No. PCT/US2012/41756 claims the benefit of U.S. Provisional Patent Application No. 61/495,879 filed Jun. 10, 2011 and entitled "INFRARED CAMERA SYSTEM ARCHITECTURES" which is hereby incorporated by reference in its entirety.
 International Patent Application No. PCT/US2012/41756 claims the benefit of U.S. Provisional Patent Application No. 61/495,888 filed Jun. 10, 2011 and entitled "INFRARED CAMERA CALIBRATION TECHNIQUES" which is hereby incorporated by reference in its entirety.
 One or more embodiments of the invention relate generally to thermal imaging devices and more particularly, for example, to the processing of thermal images.
 Conventional image processing typically requires significant processing power and substantial memory resources. In this regard, a captured image (e.g., an image frame) may include a large number of pixels, each of which may have many bits or bytes of associated data. As a result, large amounts of memory may be required to store the captured image, and operations may be required to be performed on all of the many pixels to process even a single captured image. These difficulties are compounded in realtime applications where a stream of images may need to be captured and processed without introducing significant latency or other delays.
 Unfortunately, conventional frame based approaches to image processing are often problematic. For example, a powerful processor may be required to satisfactorily process an entire image. In addition, such a processor may rely on a centralized memory system to repeatedly read and write image data to a large memory block over a shared memory bus. Such approaches can lead to processing delays and bottlenecks in the use and operation of the processor and the memory system.
 In accordance with embodiments further described herein, various techniques are provided to implement line based processing of thermal images. In one embodiment, individual lines of a thermal image frame may be provided to an image processing pipeline. Image processing operations may be performed on the individual lines in stages of the image processing pipeline.
 Various techniques are also provided to provide a flexible memory system. In one embodiment, the memory system may be used to buffer the individual lines in pipeline stages of the image processing pipeline. In another embodiment, the memory system may be used to send and receive data between various components (e.g., processing devices or other components) without relying on a single shared bus between the components. For example, data transfers may be performed between different components and different memories of the memory system using a switch fabric to route data over different buses simultaneously or substantially simultaneously.
 In another embodiment, the memory system may be implemented to support data transfers using different clocks of the various components, without requiring the components and the memory system to all be synchronized to the same clock source.
 In another embodiment, a method includes receiving a thermal image frame comprising a plurality of individual lines, wherein each individual line comprises substantially an entire row or column of thermal image data captured by a plurality of infrared sensors; providing each individual line of the thermal image frame to a line based image processing pipeline; performing image processing operations on the individual lines in stages of the image processing pipeline; and buffering the individual lines in the pipeline stages.
 In another embodiment, an infrared imaging module includes a sensor input block adapted to receive a thermal image frame comprising a plurality of individual lines, wherein each individual line comprises substantially an entire row or column of thermal image data captured by a plurality of infrared sensors; a processing device comprising a line based image processing pipeline adapted to perform image processing operations on the individual lines in stages of the image processing pipeline; and a memory system adapted to buffer the individual lines in the pipeline stages.
 In another embodiment, a device includes a plurality of components adapted to transfer thermal image data; a plurality of buses connected to the components; and a memory system connected to the components by the buses, the memory system comprising a plurality of memory buffers, each memory buffer comprising: a memory block comprising a single interface adapted to support a single read or write operation at a time, a plurality of ports, wherein each port is adapted to communicate with a corresponding one of the components over a corresponding one of the buses, and a switch fabric block adapted to selectively couple one of the ports with the memory block to permit transfer of at least a portion of the thermal image data between the corresponding one of the components and the memory block through the coupled port and the single interface.
 In another embodiment, a method of operating a memory system and a plurality of components connected to the memory system by corresponding buses includes operating a switch fabric to couple a first port of a memory buffer with a memory block of the memory buffer, wherein the memory block comprises a single interface adapted to support a single read or write operation at a time, wherein the memory buffer is part of the memory system; transferring first thermal image data between a first one of the components and the memory block over a first one of the buses, through the first port, and through the single interface; subsequently operating the switch fabric to couple a second port of the memory buffer with the memory block; and transferring second thermal image data between a second one of the components and the memory block over a second one of the buses, through the second port, and through the single interface.
 FIG. 12 illustrates a block diagram of a processing module in accordance with an embodiment of the disclosure.
 FIG. 13 illustrates a block diagram of a main memory slice (MMS) in accordance with an embodiment of the disclosure.
 FIG. 14 illustrates a block diagram of a virtual line buffer (VLB) in accordance with an embodiment of the disclosure.
 FIG. 15 illustrates a block diagram of a portion of a memory system in communication with various components in accordance with an embodiment of the disclosure.
 FIGS. 16A-E illustrate various operations performed by a memory system and a processing device in accordance with embodiments of the disclosure.
 Other connections may be used in other embodiments. For example, in one embodiment, sensor assembly 128 may be attached to processing module 160 through a ceramic board that connects to sensor assembly 128 by wire bonds and to processing module 160 by a ball grid array (BGA). In another embodiment, sensor assembly 128 may be mounted directly on a rigid flexible board and electrically connected with wire bonds, and processing module 160 may be mounted and connected to the rigid flexible board with wire bonds or a BOA.
 Socket 104 may include a cavity 106 configured to receive infrared imaging module 100 (e.g. as shown in the assembled view of FIG. 2). Infrared imaging module 100 and/or socket 104 may include appropriate tabs, arms, pins, fasteners, or any other appropriate engagement members which may be used to secure infrared imaging module 100 to or within socket 104 using friction, tension, adhesion, and/or any other appropriate manner. Socket 104 may include engagement members 107 that may engage surfaces 109 of housing 120 when infrared imaging module 100 is inserted into a cavity 106 of socket 104. Other types of engagement members may be used in other embodiments.
 Referring again to FIG. 5, the updated row and column FPN terms determined in block 550 are stored (block 552) and applied (block 555, e.g., full terms and/or estimates may be applied) to the blurred image frame provided in block 545. After these terms are applied, some of the spatial row and column FPN in the blurred image frame may be reduced. However, because such terms are applied generally to rows and columns, additional FPN may remain such as spatially uncorrelated FPN associated with pixel to pixel drift or other causes. Neighborhoods of spatially correlated FPN may also remain which may not be directly associated with individual rows and columns. Accordingly, further processing may be performed as discussed below to determine NUC terms.
 In one embodiment, the risk of introducing scene information into the NUC terms can be further reduced by applying some amount of temporal damping to the NUC term determination process. For example, a temporal damping factor 2%, between 0 and 1 may be chosen such that the new NUC term (NUCNEW) stored is a weighted average of the old NUC term (NUCOLD) and the estimated updated NUC term (NUCUPDATE). In one embodiment, this can be expressed as NUCNEW=λNUCOLD+λ)(NUCOLD+NUCUPDATE).
 Image frames 802 proceed through pipeline 800 where they are adjusted by various terms, temporally filtered, and additionally processed. In blocks 810 and 814, factory gain terms 812 (e.g., gain offsets/coefficients in one embodiment) and factory offset terms 816 (e.g., pixel offsets/coefficients and LaGrange offsets/coefficients in one embodiment) are applied to image frames 802 to compensate for gain and offset differences, respectively, between the various infrared sensors 132 and/or other components of infrared imaging module 100 determined during manufacturing and testing.
 Referring again to FIG. 8, image frame 802e may be passed to additional blocks 827-832 for further processing to provide a result image frame 833 that may be used by host device 102 as desired. In one embodiment, such processing may include: bad pixel replacement processing in block 827 (e.g., to compensate for malfunctioning or inoperative pixels); distortion correction processing in block 828 (e.g., to compensate for possible lens distortion or other distortion); video polarity processing in block 829; gamma correction processing in block 830; automatic gain compensation processing in block 831; and pseudo-color processing in block 832 (e.g., using a look up table (LUT)).
 As shown in FIG. 8, row and column FPN terms 824 and 820 and NUC terms 817 may be determined and applied in an iterative fashion such that updated terms are determined using image frames 802 to which previous terms have already been applied. As a result, the overall process of FIG. 8 may repeatedly update and apply such terms to continuously reduce the noise in image frames 833 to be used by host device 102.
 In accordance with additional embodiments, various techniques are provided to implement line based processing of thermal images. In one embodiment, individual lines of a thermal image frame may be provided to an image processing pipeline provided, for example, by one or more processing devices of infrared imaging module 100. Image processing operations may be performed on the individual lines in stages of the image processing pipeline.
 In accordance with additional embodiments, techniques are provided to implement a flexible memory system to support one or more processing devices of infrared imaging module 100. For example, in some embodiments, the memory system may be used to support line based (e.g., row based or column based) processing of thermal images provided by infrared sensor assembly 128. In this regard, a line may generally refer to a row or a column, and the terms line, row, and column are used interchangeably herein.
 Also in some embodiments, the memory system may be used to send and receive data between various components without relying on a single shared bus between the components. For example, in some embodiments, data transfers may be performed between different components and different memories of the memory system using a switch fabric to route data over different buses simultaneously or substantially simultaneously. In some embodiments, the memory system may be implemented to support data transfers using different clocks of the various components, without requiring the components and the memory system to all be synchronized to the same clock source.
 These and other features of various embodiments of the memory system are further described with regard to FIGS. 12-16E. Although the memory system will be primarily described with regard to processing module 160, it may be used with any processing device (e.g., processing module 160, processor 195, and/or other devices) and the various components of processing module 160 may be implemented as any type of processing device, logic, and/or circuitry as appropriate in various implementations.
 FIG. 12 illustrates a block diagram of processing module 160 in accordance with an embodiment of the disclosure. In the illustrated embodiment, processing module includes a sensor input block 1201, a digital signal processing (DSP) core 1206, a video output block 1207, a one time programmable (OTP) memory 1212, an inter-integrated circuit (I2C) interface 1214, a general purpose CPU 1216, a system control block 1220, a system support block 1222, a memory management unit (MMU) 1230, and memory blocks 1240.
 Sensor input block 1201 receives thermal image frames from infrared sensor assembly 128 which correspond to thermal images captured by infrared sensors 132. In this regard, sensor input block 1201 includes an FPA interface 1202 to receive the thermal image frames and an integration block 1204 to integrate the thermal image frames (e.g., provided in form of analog or digital signals) over multiple thermal image frames to provide, for example, thermal image frames with improved signal to noise characteristics.
 DSP core 1206 may be used to perform any of the various operations described herein to process thermal image data and/or related operations. For example, in some embodiments, DSP core 1206 may be used to implement pipeline 800 of FIG. 8. DSP core 1206 may be implemented as any appropriate processing device.
 Video output block 1207 provides processed thermal image frames from processing module to other components of infrared imaging module 100, host device 102, and/or other devices through one or more mobile industry processor interfaces (MIPI) 1208 and/or one or more serial peripheral interfaces (SPI) 1210.
 OTP memory 1212 (e.g., including one or more OTP memory blocks) may be implemented with appropriate control circuitry and non-volatile memory used to store various previously determined data for use in the various operations described herein. Such data may include, but need not be limited to, factory gain terms 812 and factory offset terms 816 (e.g., collectively and separately referred to as factory calibration terms), bad pixel maps/lists, pixel deltas and weights for distortion correction, look up tables (LUT) for color, polarity, and gamma corrections, and/or other data as appropriate. Although OTP memory 1212 is generally referred to herein, any type of volatile or non-volatile memory may be used in addition to and/or instead of OTP memory 1212 as may be appropriate in various implementations.
 I2C interface 1214 may be used to support communications between processing module 160 and host device 102. General purpose CPU 1216 (e.g., also referred to as a GPP core) may be used to perform various operations to support the overall operation of processing module 160 including, for example, servicing communications and commands over I2C interface 1214, initializing data and DSP core 1206, power management, write sequencing for OTP memory block 1212, direct memory access (DMA) operations, memory mapping and configuration, configuration of DSP core 1206, configuration of video output block 1207, calibration support, and/or other operations. GPP core 1216 may be implemented as any appropriate processing device.
 System control block 1220 (e.g., also referred to as a PPM block) may be used to generate clocks and reset signals, and provide on-chip power control. System support block 1222 may be used to perform various die test and CPU debugging operations, and provide one or more joint test action group (JTAG) interfaces (e.g., IEEE 1149.1 standard test access port and boundary-scan architecture).
 Memory management unit (MMU) 1230 (e.g., a memory controller) may be used to manage data communications between OTP memory 1212, memory blocks 1240, and other components of processing module 160. For example, in some embodiments, MMU 1230 may pass data over various buses 1260 connected to MMU 1230 as illustrated in FIG. 12. Thus, OTP memory 1212, MMU 1230, and memory blocks 1240 may effectively provide a memory system 1250 to support line based processing of thermal images and various operations described herein. In this regard, memory system 1250 may provide a switch fabric to selectively route data between components of processing module 160 and the various memories of memory system 1250. Such a switch fabric may be provided, for example, by MMU 1230 and/or additional circuitry of OTP memory 1212 and/or memory blocks 1240.
 Memory blocks 1240 may be implemented, for example, as volatile static random access memory (SRAM) including main memory slices (MMSs) 1242 and virtual line buffers (VLBs) 1244 further described herein. In one embodiment, memory blocks 1240 may include 3 Mbits of SRAM, with approximately 2.5 Mbits allocated to MMSs 1242 (e.g., each MMS 1242 may include approximately 256 kbits of SRAM) and approximately 0.5 Mbits allocated to all of VLBs 1244 (e.g., each of 64 VLBs 1244 may include approximately 8 kbits of SRAM). Other implementations and configurations of volatile and/or non-volatile memory are also contemplated.
 FIG. 13 illustrates a block diagram of an MMS 1242 of memory system 1250 in accordance with an embodiment of the disclosure. MMS 1242 includes a memory block 1302 and access circuitry 1304 providing a switch fabric for MMS 1242. In one embodiment, memory block 1302 may be implemented as a 256 kbit SRAM memory block having a single interface 1306 (e.g., also referred to as an SRAM access port) adapted to support a single read or write operation at a time. Access circuitry 1304 may be used to permit one or more components of processing module 160 to access memory block 1302 through multiple ports 1312 and 1314 (e.g., memory bus ports). Components of processing module 160 may also be referred to as owners of MMS 1242 when MMS 1242 is configured for access by such components. Access circuitry 1304 includes a multiplexer 1308, a port select input 1310, and ports 1312 and 1314 (e.g., each port providing read and write access in one embodiment). In some embodiments, access circuitry 1304 may be implemented with additional circuitry and/or logic as appropriate.
 In one embodiment, MMU 1230 and access circuitry 1304 may be used to permit simultaneous or substantially simultaneous access to memory block 1302 by multiple components of processing module 160. In this regard, MMU 1230 may be connected to port select input 1310 and ports 1312 and 1314 to manage access to memory block 1302 by other components of processing module 160. For example, MMU 1230 may pass data between any of the various components of processing module 160 and memory block 1302 through ports 1312 and 1314.
 FIG. 14 illustrates a block diagram of a VLB 1244 of memory system 1250 in accordance with an embodiment of the disclosure. VLB 1244 includes a memory block 1402 and a switch fabric 1404. In one embodiment, memory block 1402 may be implemented as an 8 kbit SRAM memory block having a single interface 1403 (e.g., also referred to as an SRAM access port and shown in FIG. 15 as interfaces 1403A-D for memory blocks 1402A-D of memory buffers 1244A-D) adapted to support a single read or write operation at a time. Switch fabric 1404 may be used to permit one or more components of processing module 160 to access memory block 1402 through multiple ports 1420 (e.g., memory bus ports labeled MB--0 through MB--3). Although four ports 1420 are shown in FIG. 14, any desired number of ports 1420 may be provided. Similar to MMS 1242, components of processing module 160 may also be referred to as owners of VLB 1244 when VLB 1244 is configured for access by such components.
 In FIG. 14, switch fabric 1404 is represented by arrows and may also include one or more multiplexers 1431 used to select between ports 1420. VLB 1244 may also include one or more state machines 1430, clock switches 1432, and/or other circuits 1433, any of which may be provided as part of, or separate from, switch fabric 1404. VLB 1244 receives control signals 1408, for example, from MMU 1230 which cause VLB 1244 to be configured to permit various components of processing module 160 to access memory block 1402. In one embodiment, a state machine 1430 may operate multiplexers 1431 in response to control signals 1408 to selectively couple memory block 1402 with one or more ports 1420. In this regard, in one embodiment, control signals 1408 may be used to: define the owner of VLB 1244 (signal Def_Owner); identify the next owner of VLB 1244 (signal New_Owner); switch to the next owner through a signal transition (signal Switch_Owner); and provide a clock for state machine 1430 (signal MMU_CLK). VLB 1244 also provides control signal 1410 (signal Cur_Owner) to identify the current owner of VLB 1244.
 Each port 1420 may receive corresponding input signals 1412 from a particular component of processing module 160 in communication with VLB 1244 through buses 1260 and/or MMU 1230. In one embodiment, input signals 1412 may include: a clock signal (signal MBx_CLK); an address signal (signal MBx_ADDR); a read signal (signal MBx_RD); a write signal (signal MBx_WR); and one or more additional signals (signal MBx_).
 Memory block 1402 may be synchronized by the particular clock signal MBx_CLK of the particular port 1420 to which memory block 1402 is currently connected. For example, while memory block 1402 is associated with port MB--0, memory block 1402 may be synchronized by clock signal MB0_CLK. If memory block is subsequently associated with port MB--1 (e.g., based on control signals 1408), then memory block 1402 may be synchronized by clock signal MB1_CLK. In this regard, one or more clock switches 1432 may be used to permit such changes in clock synchronization. For example, one or more clock switches 1432 may be used to provide glitchless transitions between different clock signals provided to memory block 1402. Accordingly, ownership of memory block 1402 may be rapidly switched between various components of processing module 160 running independent clocks and without requiring clock resampling or resynchronization across different clock domains (e.g., which could otherwise result in extra indeterminate latency clock cycles lost for each clock domain crossing). In one embodiment, if no clock signal is available for a selected port 1420, then memory block 1402 may be synchronized by clock signal MMU_CLK of control signals 1408.
 Each port 1420 may exchange data signals 1414 (signals MBx_D) with its associated component of processing module 160 through buses 1260 and/or MMU 1230. In some embodiments, data signals 1414 of different ports 1420 may be used simultaneously. For example, in one embodiment, data may be passed from memory block 1402 to several components of processing module 160.
 FIG. 15 illustrates a block diagram of a portion of memory system 1250 in communication with various components of processing module 160 in accordance with an embodiment of the disclosure. As shown, memory system 1250 is in communication with sensor interface 1201, DSP core 1206, video interface 1207, and GPP core 1216 over buses 1260. However, such communication may be provided with fewer or greater numbers of components of processing module 160 or other devices in other embodiments.
 MMU 1230 manages the passing of input signals 1412 and data signals 1414 between buses 1260 and VLBs 1244 (e.g. labeled 1244A-D in FIG. 15) through one or more communication channels 1510 (e.g., hardware circuit paths, multiplexed communications, and/or other forms as appropriate), some of which are shown with specific paths and others are shown more generally as a cloud in FIG. 15. In this regard, MMU 1230 may be implemented, for example, with a memory controller 1502 and/or other circuitry as appropriate to generate control signals 1408 and to control communication channels 1510.
 For example, in some embodiments, memory controller 1502 may be configured with a predetermined instruction sequence (e.g., provided on a non-transitory machine readable medium) that is optimized for pipeline 800 and/or the operations shown in FIGS. 16A-E. In this regard, memory controller 1502 may generate control signals 1408 to independently operate each of switch fabric blocks 1404A-D to support data transfers between various components for thermal image processing and/or other operations.
 As discussed, each VLB 1244 may be implemented with a memory block 1402 (e.g., labeled 1402A-D in FIG. 15) and a switch fabric 1404 (e.g., labeled 1404A-D in FIG. 15). Control signals 1408 provided by MMU 1230 may be used to control the switch fabric 1404 of each VLB 1244 to selectively pass input signals 1412 and data signals 1414 through particular ports 1420 (e.g., labeled 1420A-D for VLBs 1244A-D).
 In the embodiment shown in FIG. 15, a corresponding communication channel 1510 is provided between each of sensor interface 1201, DSP core 1206, video interface 1207, and GPP core 1216 and one of the ports 1420 of each VLB 1244. As a result, such components may share and pass data (e.g., any desired type of information, messages, and/or other content) between each other using memory system 1250. For example, in one embodiment, sensor interface 1201 may be an owner of SRAM 1402A of VLB 1244A and may access SRAM 1402A through the first port of VLB 1244A. It may be desired to share the data in SRAM 1402A with DSP core 1206 (e.g., for further processing). In this case, memory controller 1502 of MMU 1230 may generate control signals 1408 to operate switch fabric 1404A to select the second port of VLB 1244A. As a result, DSP core 1206 may now be the owner of SRAM 1402A and thus access SRAM 1402A to read or write data therewith.
 In another embodiment, MMU 1230 and/or switch fabric 1404 may be configured to permit multiple components to receive data signals 1414 over multiple communication channels 1510 from a single SRAM 1402. Accordingly, such an embodiment provides another flexible approach for sharing data of SRAMs 1402.
 In another embodiment, different components of processor module 160 may be connected to different combinations of VLBs 1244 (e.g., different subsets of VLBs 1244). For example, a first one of VLBs 1244 may be configured to communicate with a first subset of components over a first set of buses 1260, and a second one of VLBs 1244 may be configured to communicate with a second subset of components over a second set of buses 1260. In some embodiments, the first and second subsets of components and buses 1260 may be the same as each other. In other embodiments, the first and second subsets of components and buses 1260 may be different from each other (e.g., the subsets may have some or none of the same components and buses 1260 in common with each other).
 Data may be passed between one or more intermediate components and/or SRAMS 1402. For example, if communication channels 1510 are provided: between sensor interface 1201 and only VLBs 1244A-B; between DSP core 1206 and only VLBs 1244B-C; between video interface 1207 and only VLBs 1244C-D; and between GPP core 1216 and only VLB 1244D, then intermediate data passing may be performed to share data from SRAM 1402A with GPP core 1216. In this case, data in SRAM 1402A could be provided to GPP core 1216, for example, by: sensor interface 1201 reading data from SRAM 1402A and writing the data into SRAM 1402B, DSP core 1206 reading the data from SRAM 1402B and writing the data into SRAM 1402C, and video interface 1207 reading the data from SRAM 1402C and writing the data into SRAM 1402D, and GPP core 1216 reading the data from SRAM 1402D. Other data sharing combinations are also contemplated.
 FIGS. 16A-E illustrate various operations performed by memory system 1250 and DSP core 1206 in accordance with embodiments of the disclosure. For example, in some embodiments, at least some of the operations of FIGS. 16A-E may correspond to operations of pipeline 800 of FIG. 8.
 In some embodiments, the operations of FIGS. 16A-E may be performed in a line based manner. In this regard, individual lines (e.g., rows or columns) of an image frame may be passed through different pipeline stages illustrated in FIGS. 16A-E. Such an approach permits individual VLBs 1244 to be efficiently used to store data associated with a particular line of a thermal image frame. Thus, the processing of different lines of a thermal image frame may be distributed among different VLBs 1244 and different pipeline stages during the operations of FIGS. 16A-E. Such data may also be shared by various components of processing module 160 as described. Although line based processing is described primarily with regard to FIGS. 16A-E, such line based processing may be similarly implemented in any of the operations of the present disclosure as desired.
 FIGS. 16A-E identify various types of data passed between portions of memory system 1250 and DSP core 1206. In general, the various types of data identified in FIGS. 16A-E, and similar data, may be referred to as thermal image data (e.g., data that includes at least a portion of a captured thermal image, data used in the processing of thermal images, and/or any data associated with such processing).
 In one embodiment, various data identified as "OTP" may be stored by OTP memory 1212. In one embodiment, data that is less than approximately 8 kbits may be stored by VLBs 1244. Such data may be referred to as rows, row buffers, line history, LUTs, and/or other terms. Such data may be of various sizes as set forth in FIGS. 16A-E or other sizes (e.g., 60×1×8 bits, 80×1×8 bits, 80×1×16 bits, 81×1×16 bits, 1×256×8 bits, and/or other sizes). In this regard, VLBs 1244 may be used primarily to store data for individual lines (e.g., individual rows or columns) for line based processing of thermal image frames.
 In one embodiment, data that is greater than approximately 8 kbits and less than approximately 256 kbits may be stored by MSSs 1242. Such data may be referred to as full frame buffers, double full frame buffers, histograms, and/or other terms. Such data may be of various sizes as set forth in FIGS. 16A-E or other sizes (e.g., 80×60×8 bits, 80×60×16 bits, 80×60×32 bits, 1×2048×8 bits, and/or other sizes). In this regard, MSSs 1244 may be used primarily to store data for one or more thermal image frames (e.g., MMSs 1244 may be used as full frame buffers in some embodiments).
 Referring now to FIG. 16A, data 1602 includes an image frame 802 (e.g., a thermal image frame) received from infrared sensor assembly 128 for processing in one or more pipeline stages corresponding to blocks 810, 814, 580, 818, and 822 of FIG. 8. As shown, data 1602 may be received as a full image frame 802 and provided on a line by line basis through a VLB 1244 for processing. In some embodiments, image frame 802 may be approximately 80 by 60 pixels, and individual lines may be approximately 80 pixels or fewer, or approximately 60 pixels or fewer.
 Data 1604 includes factory gain terms 812, factory offset terms 816 (e.g., including pixel offsets/coefficients and LaGrange offsets/coefficients in one embodiment), NUC terms 817, column FPN terms 820, and row FPN terms 824 received from various memories and/or provided in accordance with various processing as shown.
 In some embodiments, a plurality of NUC terms 817 may be determined for an unblurred thermal image frame from an intentionally blurred thermal image frame. NUC terms 817 associated with a single one of the individual lines may be stored in a VLB 1244 and applied to the individual lines.
 In some embodiments, a plurality of factory calibration terms 812/816 may be read from OTP memory block 1212. Factory calibration terms 812/816 associated with a single one of the individual lines may be stored in a VLB 1244 and applied to the individual lines.
 An individually processed line is provided as data 1606 stored in a VLB 1244 operating as a pipeline register to be used for LaGrange calculations in one or more pipeline stages corresponding to block 814. Following block 814, processed line data (e.g., adjusted by offsets and coefficients) is passed to FIG. 16B.
 Referring now to FIG. 16B, processed line data 1620 received from FIG. 16A is stored in a VLB 1244 operating as a pipeline register and is provided for temporal filtering operations in one or more pipeline stages corresponding to block 826. Block 826 uses data 1630 which includes various lines and image frames that have been buffered, accumulated, and/or otherwise stored as shown. In particular, data 1630 may include multiple image frames that are accumulated to provide a blurred image frame in an accumulation frame buffer, individual lines of the current and accumulated image frames, local (e.g., adjacent) lines, a previous image frame, temporary lines, lines from multiple previous frames, and local line changes from previous image frames. Accordingly, in some embodiments, at least some of data 1630 corresponds to various data referenced in FIGS. 8-10.
 In some embodiments, a current thermal image frame may be compared with a previous thermal image frame to determine whether a scene has changed in block 826. For example, a first set of individual lines may be buffered in a first set of corresponding memory buffers (e.g., local pixel history line buffers in one embodiment) that correspond to a subset of the current thermal image frame. A second set of individual lines may be buffered in a second set of corresponding memory buffers (e.g., previous frame line buffers in one embodiment) that correspond to a subset of a previous thermal image frame. The first and second sets of buffered individual lines may be compared to determine if a scene has changed (e.g., changes between the buffered lines of the thermal image frames may be buffered using one or more local kernel line buffers in one embodiment). If the scene has changed, then the current and/or previous thermal image frames may be accumulated (e.g., in a blur-frame accumulation frame buffer in one embodiment).
 Following block 826, processed line data 1638 (e.g., temporally filtered line data) is stored in a VLB 1244 operating as a pipeline register and is provided for bad pixel replacement operations in one or more pipeline stages corresponding to block 827. Data 1640 includes bad pixels maps/lists provided by OTP memory 1212 as well as various lines that have been buffered, accumulated, and/or otherwise stored for processing in block 827 as shown. In particular, data 1640 may include local lines, a bad pixel map provided as a full frame and temporary lines, and local lines containing the bad pixel map.
 In some embodiments, one or more pixels of a thermal image frame may be replaced in block 827. For example, a first set of individual lines may be buffered in a first set of corresponding memory buffers (e.g., local kernel line buffers) that correspond to a subset of the thermal image frame. A second set of individual lines may be buffered in a second set of corresponding memory buffers (e.g., local line buffers) that correspond to a subset of a pixel map. One or more pixels of the first set of individual lines may be replaced based on the second set of individual lines. Following block 827, processed line data (e.g., with bad pixels replaced) is passed to FIG. 16C.
 Referring now to FIG. 16C, processed line data 1650 received from FIG. 16B is stored in a VLB 1244 operating as a pipeline register and is provided for distortion processing operations (e.g., to compensate for warping or distortion effects caused by one or more optical elements and/or other sources) in one or more pipeline stages corresponding to block 828. Data 1655 includes various pixel deltas and weights, and bad pixel maps provided by OTP memory 1212 (e.g., collectively and separately referred to as distortion correction terms). In particular, data 1655 may include pixel deltas, pixel weights, and bad pixel maps buffered in full frames and in individual lines.
 In some embodiments, individual lines of a thermal image frame may be corrected to compensate for distortion. For example, a plurality of distortion correction terms may be read from OTP memory block 1212. Distortion correction terms associated with a single one of the individual lines may be stored in a VLB 1244 and applied to the individual lines.
 Following block 828, processed line data 1660 (e.g., de-warped line data) is stored in a VLB 1244 operating as a pipeline register and is provided for video polarity processing, gamma correction processing, automatic gain compensation processing, and pseudo-color processing in one or more pipeline stages corresponding to blocks 829, 830, 831, and 832. Data 1665 includes various LUT information provided by OTP memory 1212, histograms, and calculated values as shown.
 In some embodiments, LUT data may be used in the processing performed in blocks 829, 830, 831, and 832. For example, LUT data may be read from OTP memory block 1212. LUT data associated with a single one of the individual lines may be stored in a VLB 1244 and applied to the individual lines. Following blocks 829, 830, 831, and 832, processed line data (e.g., processed by blocks 829, 830, 831, and 832) for a result image frame 833 (e.g., a result thermal image frame) is provided and stored in a VLB 1244 operating as a pipeline register.
 Referring now to FIG. 16D, data 1670 includes accumulated thermal image frames (e.g., provided in accordance with block 535) received as full thermal image frames and provided as individual lines for processing in one or more pipeline stages corresponding to blocks 540 and 545. In particular, data 1670 may include multiple image frames that are accumulated to provide a blurred image frame in an accumulation frame buffer and individual lines of the accumulated image frames.
 Following blocks 540/545, processed line data 1672 (e.g., blurred averaged accumulated lines) is stored in various VLBs 1244 operating as pipeline registers and multiple lines are buffered as a full image frame in an MMS 1242. Data 1672 is provided for processing in one or more pipeline stages corresponding to blocks 550 and 555.
 Block 550 also uses data 1674 which includes previous line data. Block 550 also provides column FPN terms 820 and row FPN terms 824 as part of data 1674 which is provided to FIG. 16A as shown. Block 550 also provides column noise estimates 1676 and row noise estimates 1678 to block 555 which are stored in VLBs 1244 operating as pipeline registers.
 In some embodiments, a previous thermal image frame may be used to determine column FPN terms 820 and row FPN terms 824. For example, a set of individual lines may be buffered in a set of corresponding memory buffers that correspond to a subset of the previous thermal image frame. The lines of the previous thermal image frame may be processed to determine row and column noise terms for the current thermal image frame.
 Block 555 uses data 1672, 1676, and 1678 to provide processed line data 1680 (e.g., with column and row noise removed) which is stored in a VLB 1244 operating as a pipeline register. Blocks 560, 565, 570, 571, 572, 573, and 575 use data 1680 and 1682 (e.g., including buffered line data and NUC terms 817) and operate as one or more pipeline stages to provide NUC terms 817 to FIG. 16A as shown. In particular, data 1682 may include individual lines of NUC terms 817.
 Referring now to FIG. 16E, further operations of block 831 are identified. In this regard, block 831 uses data 1690 that includes histogram and LUT information as shown for processing in one or more pipeline stages and to communicate such data with FIG. 16C as shown.
 In view of the present disclosure, it will be appreciated that line based processing of thermal image frames may be performed in an efficient pipelined manner, thus distributing processing tasks into smaller tasks than conventional frame based processing. Such line based processing may be efficiently used with hardware and software optimized for fast and efficient processing, such as relatively small memory buffers with single interface memory blocks may be used with associated switch fabric circuitry to support efficient line based processing.
 In this regard, memory system 1250 may be efficiently and flexibly utilized by processing module 160 to provide individual components with memory access through associated buses 1260, rather than a conventional main memory system shared system bus. In addition, by passing clock signals from individual components over buses 1260, individual memory blocks of memory system 1260 may be separately synchronized with their associated components (e.g., owners).
 In addition, as discussed, memory system 1250 may be used to transfer (e.g., move, copy, and/or otherwise pass) and share data between individual components without requiring a centralized memory system to perform such transfers through conventional read and write commands and/or direct memory access (DMA) engines. For example, control signals 1408 may be used to change the particular port 1420 of a VLB 1244, thus effectively transferring the data of its associated memory block 1402 to be used by a different component associated with a different port 1420 of the same VLB 1244. As discussed, data may also be transferred to different memory blocks 1402 my individual components as desired.
 Other embodiments are also contemplated. In some embodiments, various system implementations described herein may be scaled for increased capacity and/or performance as desired for various implementations. For example, it is contemplated that one or more additional components (e.g., local or remote to processing module 160) may be networked or otherwise interfaced with memory system 1260 and/or processing module 160 to use the various features thereof. It is also contemplated that the various components of memory system 1260 may be may be scaled to accommodate such additional components and/or to improve performance.

References: Application No. 61
 Application No. 61
 Application No. 61
 Application No. 61
 Application No. 61
 Application No. 61