Patent Description:
There are a number of different configurations for x-ray CT systems. In x-ray microscopy systems, because the x-ray sources and detectors are large and the objects being scanned are typically small, the x-ray sources and detectors are largely fixed, while the objects are rotated in the x-ray beam. This contrasts with medical CT systems where the patient is fixed and the sources and detectors rotate around the patient. <CIT> describes a particle beam microscope implementing collision avoidance, wherein a surface model of a specimen chamber is calculated from images taken of the specimen chamber without an object contained in the specimen chamber and another surface model of an object is calculated based on images taken in a load-lock chamber and the object is loaded to and moved in the specimen chamber based on the surface model and the another surface model.

In many cases, the objects scanned in X-ray microscopy systems have a priori unknown shapes. Even in the cases where a CAD model is available or the object is from a dimensionally-known core-sample, for example, the exact alignment of the object is often unknown. Moreover, the alignment may be changed when different regions of interest are selected and the object is realigned in the beam path. This leads to the problem that while the object is moved to be scanned (mostly rotated) the object might collide with the scanning setup (the parts of the X-ray-source or detector that are most proximate to the object). The challenge of avoiding collisions is often made more difficult by the fact that the X-ray source and/or the detector will need to be moved into close proximity to the object for optimal system performance.

Similar setups also exist in other microscopy/tomography systems operating in other regions of the electromagnetic spectrum such as optical coherence tomography and confocal microscopy (optical projection tomography). Still other examples include scanning electron microscopes (SEMs) and focused ion beam (FIB) systems - i.e. charged particle imaging systems.

Collisions can be avoided if a 3D model (e.g. a 3D representation such as a mesh or surface data) of the object and a 3D model of the setup is available. Another use of such a digital 3D model of the object is the prediction of the resulting X-ray images, through ray tracing. Such simulated images can be used for region of imaging selection, contrast enhancement, system configuration (e.g. source voltage or power) or fully automated scanning as well as an improved X-ray reconstruction.

In general, according to one aspect, the invention features a collision avoidance system for a microscope. This system comprises a camera or multiple cameras for capturing images of an object loaded into the microscope and a computer that processes image data of an object at different angles from the camera, generates a model of the object, and uses the model to configure the microscope for operation.

In embodiments, the computer uses the model to avoid collisions between the microscope and the object.

Also, the microscope might be an x-ray microscope, scanning electron microscope, a focused ion beam system, or an optical microscope, for example.

Preferably, the computer further has a model of a source subsystem and/or a detector subsystem, which are also used to configure the microscope. In addition, the computer receives current position data of a source subsystem and/or a detector subsystem in examples. Further, the computer might render a display including a model of the object.

In some embodiments, a light source is provided for illuminating the object under control of the computer. The light source might illuminate the object in different colors and/or a background of the object.

In general, according to another aspect, the invention features a collision avoidance method for a microscope. The method comprises capturing image data of an object at different angles, generating a model of the object, and using the model to configure the microscope for operation.

In general, according to another aspect, the invention features a user interface rendered on a display of a microscopy system. This interface comprises controls for moving a source stage and/or object stages and/or a detector stage and an image region in which a model of an object to be imaged is rendered.

The above and other features of the invention including various novel details of construction and combinations of parts, and other advantages, will now be more particularly described with reference to the accompanying drawings and pointed out in the claims. It will be understood that the particular method and device embodying the invention are shown by way of illustration and not as a limitation of the invention.

<FIG> is a schematic diagram of a microscopy system <NUM> to which the present invention is applicable.

The illustrated microscopy system <NUM> is an X-ray CT system and generally includes several subsystems. An X-ray source subsystem <NUM> generates a polychromatic or possibly monochromatic X-ray beam <NUM>. An object stage subsystem <NUM> with object holder <NUM> holds an object <NUM> in the beam and positions and repositions it to enable scanning of the object <NUM> in the stationary beam <NUM>, <NUM>. A detector subsystem <NUM> detects the beam <NUM> after it has been modulated by the object. A base, such as a platform or optics table <NUM>, provides a stable foundation for the microscopy system <NUM> and its subsystems.

In general, the object stage subsystem <NUM> has the ability to position and rotate the object <NUM> in the beam <NUM>. Thus, the object stage subsystem <NUM> will typically include linear and rotation stages. The illustrated example has a precision <NUM>-axis stage <NUM> that translates and positions the object along the x, y, and z axes, very precisely but only over relatively small ranges of travel. This allows a region of interest of the object <NUM> to be located within the beam <NUM>/<NUM>. The <NUM>-axis stage <NUM> is mounted on a theta stage <NUM> that rotates the object <NUM> in the beam around the y-axis. The theta stage <NUM> is in turn mounted on the base <NUM>.

The source subsystem <NUM> will typically be either a synchrotron x-ray radiation source or alternatively a "laboratory x-ray source" in some embodiments.

As used herein, a "laboratory x-ray source" is any suitable source of x-rays that is not a synchrotron x-ray radiation source. Laboratory x-ray source <NUM> can be an X-ray tube, in which electrons are accelerated in a vacuum by an electric field and shot into a target piece of metal, with x-rays being emitted as the electrons decelerate in the metal. Typically, such sources produce a continuous spectrum of background x-rays combined with sharp peaks in intensity at certain energies that derive from the characteristic lines of the selected target, depending on the type of metal target used. Furthermore, the x-ray beams are divergent and lack spatial and temporal coherence.

In one example, source subsystem <NUM> is a rotating anode type or microfocused source, with a Tungsten target. Targets that include Molybdenum, Gold, Platinum, Silver or Copper also can be employed. Preferably a transmission configuration is used in which the electron beam strikes the thin target from its backside. The x-rays emitted from the other side of the target are used as the beam <NUM>.

The x-ray beam generated by source subsystem <NUM> is often conditioned to suppress unwanted energies or wavelengths of radiation. For example, undesired wavelengths present in the beam are eliminated or attenuated, using, for instance, energy filters (designed to select a desired x-ray energy range (bandwidth)) held in a filter wheel <NUM>. Conditioning is also often provided by collimators or condensers.

For other types of microscopy systems, different sources would be used. For example, if the microscopy system <NUM> is a scanning electron microscope (SEM), the source subsystem <NUM> generates an electron beam. If the microscopy system is a focused ion beam (FIB) system, then the source subsystem <NUM> generates an ion beam. Finally, if the microscopy system operates in the optical regime, then the source subsystem <NUM> generates a beam of light.

When the object <NUM> is exposed to the X-ray or other beam <NUM>, the X-ray photons or particles, which propagate through the object, form a modulated beam <NUM> that is received by the detector subsystem <NUM>. In some other examples, an objective lens is used to form an image onto the detector subsystem <NUM> of the microscopy system <NUM>.

Typically, a magnified projection image of the object <NUM> is formed on the detector subsystem <NUM>. The magnification is equal to the inverse ratio of the source-to-object distance <NUM> and the source-to-detector distance <NUM>.

To achieve high resolution, the X-ray CT system <NUM> utilizes a very high resolution detector subsystem <NUM> in conjunction possibly with positioning the object <NUM> close to the X-ray source subsystem <NUM>. In this case, the resolution of the x-ray image is limited by the resolution of the detector subsystem <NUM>, the focus spot size of the X-ray source subsystem <NUM>, the position of the object <NUM> and the geometrical magnification of the object <NUM> at the detector subsystem <NUM>. In other examples, the object might be positioned close to the detector subsystem, especially when the detector subsystem employs optical magnification.

For other types of microscopy systems, the appropriate detector (electron, particle, optical) is selected.

Typically, the source subsystem <NUM> and the detector subsystem <NUM> are mounted on respective z-axis stages. For example, in the illustrated example, the source subsystem <NUM> is mounted to the base <NUM> via a source stage <NUM>, and the detector subsystem <NUM> is mounted to the base <NUM> via a detector stage <NUM>. In practice, the source stage <NUM> and the detector stage <NUM> are lower precision, high travel-range stages that allow the source subsystem <NUM> and the detector subsystem <NUM> to be moved into position, often very close to the object during scanning and then be retracted to allow the object to be removed from, a new object to be loaded onto, and/or the object to be repositioned on the object holder <NUM> of the object stage subsystem <NUM>.

The operation of the microscopy system <NUM> and the scanning of the object <NUM> is controlled by a computer subsystem <NUM> that often includes an image processor <NUM> and a controller <NUM>.

The computer system <NUM> includes one or more processors <NUM> along with their data storage resources such as disc or solid-state drives, and memory MEM. The processors <NUM> execute an operating system <NUM> and various applications run on that operating system <NUM> to allow for user control and operation of the microscopy system <NUM>. A display device <NUM> connected to the computer subsystem <NUM> displays information from the microscopy system <NUM> such as the tomographic reconstructions in the graphical user interface. User input device(s) <NUM> such as a touch screen, computer mouse, and/or keyboard enable interaction between the operator and the computer subsystem <NUM>.

The controller <NUM> allows the computer subsystem <NUM> to control and manage components in the X-ray CT microscope <NUM> under software control. The controller might be a separate computer system adapted to handle realtime operations or an application program executing on the processor <NUM>. The source subsystem <NUM> includes a control interface <NUM> allowing for its control and monitoring by the controller <NUM>. Similarly, the object stage subsystem <NUM> and the detector subsystem <NUM> have respective control interfaces <NUM>, <NUM> for allowing for their control and monitoring by the computer subsystem <NUM> via the controller <NUM>.

To configure the microscopy system <NUM> to scan the object and to adjust other parameters such as the geometrical magnification, the operator utilizes the user interface rendered on the display device to adjust the source-to-object distance <NUM> and the source-to-detector distance <NUM> by respective operation of the source stage <NUM> and detector stage <NUM> to achieve the desired scanning setup.

Specifically, the source stage <NUM> and detector stage <NUM> include respective motor encoder systems or other actuator systems that allow the computer system <NUM> via the controller <NUM> to position the respective x-ray source subsystem <NUM> and the detector subsystem <NUM> to specified positions via the control interfaces <NUM>, <NUM>. Further, the source stage <NUM> and detector stage <NUM> signal the controller <NUM> of their actual positions.

The operator of the system under automatic control operates the object stage subsystem <NUM> to perform the CT scan via computer subsystem, the controller <NUM> and the control interfaces <NUM>, <NUM>, <NUM>. Typically, the object stage subsystem <NUM> will position the object by rotating the object about an axis that is orthogonal to the optical axis of the x-ray beam <NUM>, <NUM> by controlling the theta stage <NUM> and/or position the object in the x, y, z axes directions using stage <NUM>.

The detector subsystem <NUM> creates an image representation of the photons or particles from the attenuated beam <NUM>. Often, it includes a scintillator and an electronic spatially resolved electronic detector in the case of an X-ray system, for example. The image formed at the detector system <NUM> is also known as a projection or projection image.

Using the user interface rendered on the display device, the operator defines/selects scanning set up including the CT scan parameters. These include voltage settings that help to determine the X-ray energy spectrum and exposure time on the X-ray source subsystem <NUM>. The operator also typically selects other settings such as the field of view of the X-ray beam <NUM> incident upon the object <NUM>, the number of X-ray projection images to create for the object <NUM>. Generally, the scanning setup includes the angles to rotate and position the object by the stage subsystem <NUM>. In addition, the source-to-object distance <NUM> and the source-to-detector distance <NUM> are often specified and these are converted to the necessary positions or settings for the source stage <NUM> and detector stage <NUM> as part of the scanning setup.

The computer subsystem <NUM>, with the possible assistance of its image processor <NUM>, accepts the set of images from the detector subsystem <NUM> associated with each rotation angle of the object <NUM> to build up the scan. The image processor <NUM> combines the projection images using a CT reconstruction algorithm to create 3D tomographic volume information for the object. The reconstruction algorithm may be analytical, where convolution or frequency domain filtering of the projection data is combined with back projection onto a reconstruction grid. Alternatively, it may be iterative, where techniques from numerical linear algebra or optimization theory are used to solve a discretized version of the projection process, which may include modeling of the physical properties of the imaging system.

The present microscopy system <NUM> further adds an optical camera <NUM> such as a video camera that collects image data of the object <NUM> held in the object holder <NUM>. This camera is typically mounted directly or indirectly to the system base <NUM> via a mounting system <NUM>, such as a bracket. Typically, optical camera <NUM> collects the images in the visible portion of the spectrum and/or in the adjacent spectral regions such as the infrared. Usually, the optical camera <NUM> has a CCD or CMOS image sensor. Also included is a light source <NUM> that illuminates the object in the spectral regions employed by the optical camera. Preferably, the light source <NUM> illuminates the object <NUM> in multiple spectral regions (colors), and the optical camera <NUM> collects different color images in each of those spectral regions. In addition, the light source preferably selectively illuminates the object in one mode of operation and illuminates the background of the object in another mode of operation.

<FIG> shows the procedure performed by a collision avoidance application <NUM> executing on the computer system <NUM>.

In step <NUM>, the collision avoidance application <NUM> controls the object stage system <NUM> of the microscopy system to first rotate the object <NUM><NUM> degrees around the y-axis by control of the theta stage <NUM>. To save time, this rotation might be done continuously. While the object <NUM> turns, the optical camera <NUM> records a video and/or serial images of the object and this image data is provided to the computer subsystem such as via a USB cable, for example. The image data is stored by the collision avoidance application <NUM> of the computer subsystem <NUM>. The scan is repeated several times (twice) using different illumination settings (described later) by control of the light source <NUM>.

The current implementation uses two separate illumination conditions. The idea is to illuminate the object and the background in the first image <NUM> and only the background in the second image <NUM> by the collision avoidance application's control of the light source <NUM>.

In step <NUM> of <FIG>, the collision avoidance application <NUM> isolates the object in the image data. In one embodiment, the application analyzes the image data from the two illumination conditions. If the background appears identical in the two images, the collision avoidance application <NUM> performs a simple image subtraction to isolate the object <NUM> in the image data.

Of course, in practice it is hard to realize perfect lighting conditions to make the background identical. Therefore, in one example, the collision avoidance application <NUM> executing on the computer subsystem <NUM> uses a calibration step to match the background intensity of the two illumination conditions. After adjusting for the different background illumination strength, the unmixed image calculated by the collision avoidance application <NUM> as the difference between the foreground and the background. This calculated image isolates the object <NUM> and corresponds to a normal reflectance image of the object (in front of a now dark background). A second unmixed image is obtained by the collision avoidance application <NUM> by inverting the contrast of the transmission image and subtracting the average background intensity (can be estimated from the first unmixed image). This image corresponds to the shadow outline of the object (see <FIG>). Both images are fused together (mean) for reconstruction and filtered using a tomographic high-pass filter (for example "ram-lak" filter) by the collision avoidance application <NUM>. Alternatively, either one of those images can be used for object reconstruction, e.g. using only the background image corresponds to a shadow carving reconstruction.

In still other examples, the user via the user input device(s) <NUM>, such as a mouse and keyboard, draws bounding boxes surrounding the object at a few angles, a minimum of <NUM> orthogonal angles is required, to specify the extent of the object <NUM>. This may be used to define an outline of the object or to refine (trim or extend) the constructed 3D outline.

<FIG> shows the user interface <NUM> generated by the collision avoidance application <NUM> executing on the operating system <NUM> of the computer system <NUM> and this user interface rendered on the display device <NUM>. It includes an image region <NUM> showing the current image data received from the optical camera <NUM> when the webcam view button <NUM> is selected, which operates as a radio-button with respect to the system model button <NUM>.

The user interface <NUM> enables selection of different scanning setup parameters. An objective selector <NUM> enables dropdown selection of different objectives on a turret of the detector subsystem <NUM>. Also included is a voltage setting <NUM> and a power setting <NUM> for the source subsystem <NUM>.

The user interface <NUM> includes a source position portion <NUM>. The source position portion <NUM> includes position control buttons for operating the source stage <NUM> and a step size. It also includes a source position readout showing the current position of the source stage and thus the source subsystem <NUM>.

The user interface <NUM> includes a detector position portion <NUM>. The detector position portion <NUM> includes position control buttons for operating the detector stage <NUM> and a step size. It also includes a detector position readout showing the current position of the detector stage and thus the detector subsystem <NUM>.

The user interface <NUM> also includes an object position portion <NUM>. It also shows the position of the object by showing the x-axis, y-axis, z-axis positions of the <NUM>-axis sample stage <NUM>. It also shows the object's theta position by displaying the position of the theta stage <NUM>. There are separate position control buttons for operating each axis of the <NUM>-axis sample stage <NUM> and the theta stage <NUM> and separate step size settings.

Also shown is the user interface <NUM> generated by the computer system <NUM> and displayed on the display device <NUM> when a crosshairs overlay button <NUM> is selected, which generates the graphic crosshairs overlay <NUM> in the image region <NUM>. When Fits All overlay button <NUM> is selected a graphic box overlay is generated around the object in the image region <NUM>.

<FIG> shows the user interface generated by the computer system <NUM> and displayed on the display device for the second optical scan of the object under different lighting conditions. Here, the light source <NUM> is only lighting the background.

Returning to <FIG>, the video data from the camera <NUM> is pre-processed by collision avoidance application <NUM> in step <NUM>. The illumination conditions are unmixed to separate the foreground (the object) and the background (the rest of the camera field of view) by the collision avoidance application <NUM>.

Then in step <NUM>, the image data from the camera <NUM> is then used by collision avoidance application <NUM> to reconstruct a volumetric 3D model of the object <NUM>. In one example, the collision avoidance application executes a specially weighted filtered back projection algorithm to reconstruct the object from the image data. Often this 3D model is a shell or boundary. Such models represent the surface, i.e. the boundary of the object, not its volume. That said, in other examples, the model is a solid model that defines the volume of the object, such as one built with constructive solid geometry.

At the same time, the collision avoidance application also contains a 3D system model <NUM> of the microscopy system <NUM> and especially the portions of the microscopy system <NUM> that could collide with the sample <NUM>. Thus, generally the 3D system model <NUM> includes models of the x-ray source subsystem <NUM> and the detector subsystem <NUM>. This model could be a shell or boundary that was generated based on the image data from the camera. In other examples, it could be a solid model created as part of the design or manufacture of the system <NUM>.

In addition, the collision avoid application receives the current position <NUM> of the source stage, current position of the detector stage, the x-axis, y-axis, z-axis positions of the <NUM>-axis sample stage <NUM>, and the position of the theta stage <NUM>.

In addition, the collision avoid application receives the current selected objective for the detector subsystem via the interface <NUM> when the detector subsystem includes a turret <NUM> with multiple objectives. This information is important when the different objectives will have different clearances with respect to the object <NUM>.

After the 3D reconstruction into a voxel volume, a point cloud of the object is generated in step <NUM>. Specifically, the object is segmented. The segmentation can be done by thresholding, hysteresis thresholding, principal curves, or machine learning. The volume is then converted into the point-cloud one example.

Then, the collision avoidance application generates a 3D model of the region around the object <NUM> by combining the point cloud of the object, the 3D system model <NUM>, the current position data <NUM>, and the currently selected objective. The model is then rendered on the display device <NUM> in step <NUM> and when the model view button <NUM> is selected.

<FIG> shows the user interface <NUM> generated by the computer system <NUM> and displayed on the display device <NUM> after the reconstruction of the object. The realtime video image of the object and parts of the system <NUM> proximate to the object are replaced with the generated 3D solid model. This model includes the 3D rendered point cloud of the object <NUM>, the rendered source subsystem <NUM>, and the rendered detector subsystem <NUM>. The relative positions of the rendered object <NUM>, the rendered source subsystem <NUM>, and the rendered detector subsystem <NUM> are positioned based on the current position data <NUM> which characterized the scanning setup.

Now the reconstruction of the object and the 3D solid model is employed to configure the microscope for the x-ray scan of the object in step <NUM>.

In one example, collisions are determined by using triangle intersections between models that represent system hardware and the object. Bounding boxes for each component of the model are provided and if those bounding boxes intersect, then each triangle of the models is analyzed, in the case of STL (Standard Triangle Language) formatted models. If the bounding boxes for those triangles intersect, then each triangle is tested for intersection to conclude whether a collision is possible.

In the current embodiment, the user is limited to moving one stage or axis at a time. This movement is then interrupted when a collision is deemed to occur if the movement continued or before it is executed.

There are different occasions when collisions are tested. One of them is ahead of each move commanded by the user. And if there is a collision on that axis, the system moves the axis to the closest non-collided position and informs that user that this is the closest position.

Collisions are also tested ahead of a compounded move, such as during a beam line change on the detector. For example, when the user commands the system to change from an objective mounted on the turret to a macro lens which is a larger field of view lens mounted next to the turret, the system performs a collision test ahead of all the motor activations before allowing the switch.

In addition, the system also tests collisions right before an acquisition, such as during a tomography scan when the object is moved by the X, Y, Z and theta axes. The system tests all of those moves prior to allowing the acquisition to proceed.

<FIG> and <FIG> shows the user interface generated by the computer system <NUM> and displayed on the display device. The reconstruction of the object is employed to move the detector subsystem into position close to the object without colliding with the object as shown in <FIG>. Then the reconstruction of the object is employed to move the source subsystem into position close to the object without colliding with the object as shown in <FIG>. The system is ready to execute x-ray scan as shown in <FIG>.

In addition, with the constructed 3D solid model, optimized scan trajectories (total number of projections and / or number of projections per angle at any particular angle of rotation) are created by the collision avoidance application to reduce subsequent x-ray tomography artifacts (e.g. optimizing HART or deciding if <NUM> + fan allows for a faster scan).

Moreover, the positioning of source subsystem and detector system can be optimized to not only avoid collision but to achieve scans which are the fastest or best resolution or largest field of view.

Finally, the 3D solid model can be employed to visually guide positioning of source and detector during initial hardware setup.

As alternative to meshing, the volume can be used to calculate collisions directly (using a distance transform to the surface of the object).

The <NUM> degree rotation is required since the computer system <NUM> needs a full 3D scan of the object. This holds true for other alternative technologies like stripe projection or laser line scanning that could be employed in addition to or as an alternative to the video camera and backprojection reconstruction described above. The continuous rotation reduces the required time for such a scan significantly. The recorded images are composed of a static background and the (rotating) foreground. In order to reconstruct, the computer system <NUM> should isolate the foreground portion of the image to obtain the object. The present approach uses the images from the different lighting conditions to perform this isolation. Other approaches are possible such as generation of a background model or modeling the microscope's enclosure without the object present.

The reconstruction of the object is a (weighted) filtered back projection in one example. The computer system preferably uses a new form of weighting that is calculated from the constancy of intensity (color could be used as well) assumption that is very often used in optical flow calculations. Additional forms of weighting can be derived from a stereo (multi) camera system in which the weights are from by the constancy of intensity between the simultaneous stereo (multicamera) images. Another option is a fringe projection system.

After the FBP reconstruction the 3D volume contains the 3D representation of the object. Now this volume can be used directly to calculate collisions (using a distance transform) or to simulate the X-ray images (by filling the object with a predefined material). Alternatively, the volume is segmented (e.g. by thresholding) and the resulting point cloud is converted into a mesh representation (e.g. by triangulation, alpha shapes or a similar algorithm).

Background subtraction: Background subtraction is an unsolved problem and active field of research. There are quite a few methods in the literature. Closest to our approach are green screen techniques that use color to separate backgrounds. While their performance is good to optimal we would need at least two colors to be able to segment every type of object, and we would still have the problem of a non-uniform background (we can't change the housing (door) of the X-ray microscopes.

Background subtraction records a background image (without the object) and tries to remove (subtract) it from the final images. This has strong limitations depending on other changes in the scene (like illumination or shadows caused by moving parts). The two-illumination procedure proposed offers the following advantages:.

It should be appreciated that the present system and method described above are not limited to a (X-ray) microscope, the scan can be done by a dedicated hardware (which of course allows the inclusion of other methods). One key component that is linked to the X-ray microscope is the fact that all coordinate systems are aligned. This is achieved by using the same calibration object (alignment post) for both the X-ray and the camera system. Such a Calibration object can be a simple sphere that is visible in both systems or a dot-grid which has the same property.

While this invention has been particularly shown and described with references to preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the invention encompassed by the appended claims.

Claim 1:
A collision avoidance system for a microscopy system (<NUM>), comprising:
a camera (<NUM>) or multiple cameras for capturing images of an object (<NUM>) loaded into the microscopy system (<NUM>);
a computer (<NUM>) that processes image data of the object at different angles from the camera (<NUM>), generates model of the object (<NUM>), and uses the model to configure the microscopy system (<NUM>) for operation; and
a light source (<NUM>) for illuminating the object under control of the computer (<NUM>),
characterized in that the light source (<NUM>) is configured to illuminate the object (<NUM>) using different illumination conditions and the computer (<NUM>) is configured to use the image data of the object at different angles from the camera and from the different illumination conditions to isolate the object (<NUM>) to generate the model.