Abstract:
The present system provides a tool to estimate the relative pose of a generic object with respect to a camera view-point by processing 2D images from a monocular camera in real-time. The capability of solving the pose estimation problem relies on the robust detection and matching in consecutive image frames of significant visual features belonging to an object of interest. To accomplish this, the system incorporates 3D modeling of the object of interest. In one embodiment, the shape of interest may be approximated by a parametric surface such as a cylinder, sphere, ellipsoid, or even complex non-parametric models. The system can restrain information retrieved at a 2D image level to estimate parameters about the pose. In operation, the accuracy of the 3D pose estimation of the object is a function of the degree of approximation of the selected model and the ability to select and track relevant features across consecutive image frames.

Description:
BACKGROUND 
       [0001]    It is sometimes desired to allow a person to combine virtual reality with a real world surface. For example, a user may point the camera of a computer or smart-phone or tablet at a wall and the wall will appear on the display of the camera device. In addition, some virtual object will appear in the image as well, and appear as if it is part of the real world environment, such as, for example, a basketball hoop appearing to be affixed to the wall. The virtual content may include labels, 3D models, shading, illumination, and the like This is referred to as Augmented Reality (“AR”). In order for the view of the real world and the virtual scene to align properly (i.e. to be properly registered), the pose (i.e. 3D position and orientation) and other properties of the real and virtual cameras must be the same. 
         [0002]    Estimating the pose of an object relative to the real world is a task of an AR system. Many different AR tracking methods and systems are available in the current art, including mechanical, magnetic, ultrasonic, inertial, and vision-based, as well as hybrid methods and systems, which combine the advantages of two or more technologies. The availability of powerful processors and fast frame-grabbers has made vision-based tracking methods desirable for various purposes due to their accuracy, flexibility, and ease of use. 
         [0003]    When triggering an application in a camera on a certain surface in order to see augmented information on that surface it is possible to calculate the relative camera pose between the pose of the camera when the application was triggered to its poses in consecutive camera frames. When the information presented is 3D content it is useful to register the camera frames correctly (which are 2D by nature). Small errors in the 2D registration may be reflected in large misalignments of the 3D content. 
         [0004]    The registration process used for Augmented Reality on planar surfaces is known as planar tracking or homography tracking. In the past, planar tracking or homography tracking has been done in contexts such as aligning different patches taken from space satellites. In Augmented Reality the goal in many cases is displaying 3D content registered in real time to a real world surface or environment. One prior art approach tries to identify strong local features in the image (such as corners) and track those local features as the camera is moving to register the image. With a sizable amount of local features on the real world surface, it is possible to track the plane reliably and in real time. The local features approach can only work on surfaces that are well textured which limits the usability of the application. 
         [0005]    Another approach (sometime called the direct approach) tries to use all the pixels in the image and match between frames. The methods using the direct approach tend to be computationally intensive and are typically unable to deal with significant illumination changes. In addition, the approach has been limited in the number of degrees of freedom (DOF) that are available. 
         [0006]    Six degrees of freedom registration means the relation between the camera and the planar surface on which information is being augmented is practically the full range of motions one can expect and in particular: moving the camera up and down, left and right, forward and backward and tilting it both in rotation and skewed angles with respect to the surface being imaged. The same applies the other way around meaning moving the surface with respect to the camera. 2DOF registration accommodates only for a limited set of motions and in particular up and down and left and right. Different degrees of freedom can be defined in between these two but only 6DOF supports the full set of motions that can be done in reality. 
         [0007]    Fiducial-based vision-based tracking is popular in AR applications due to the simplicity and robustness that such tracking offers. In the prior art, fiducials are physical objects of predefined shape (and possibly size), and are usually integrated with an identification mechanism for uniquely recognizing individual fiducials. Fiducials are placed in a scene and the camera position is calculated according to their locations in the images. 
         [0008]    Another approach is called Natural-Feature Tracking (NFT). NFT methods rely on certain features found in the real world. However, the natural features that can be used should have some easily identified and somewhat unique characteristics. Thus, NFT methods limits tracking to highly-textured objects or environments in which prominent scene features can be robustly and quickly located in each frame. NFT methods usually exhibit increased computational complexity compared with fiducial-based methods, as well as reduced accuracy, since little is assumed about the environment to be tracked. NFT methods are less obtrusive and can provide more natural experiences. Nevertheless, such methods are difficult to use for creating natural user-interfaces. 
         [0009]    Planar shapes have also been used for tracking in the prior art. Ruiz et al. (hereinafter referred to as Ruiz 2006) (Alberto Ruiz, Pedro E. Lo{acute over (p)}ez de Teruel and Lorenzo Fernández, “Robust Homography Estimation from Planar Contours Based on Convexity”, European Conference on Computer Vision, pp. 107-120, 2006.) proposed a projective approach for estimating the 3D pose of shape contours. An invariant-based frame construction is used for extracting projective invariant features from an imaged contour. The features are used for constructing a linear system of equations in homogeneous coordinates that yields the camera pose. Although theoretically general, the construction proposed in Ruiz 2006 limits the scope of usable shapes by several assumptions on shape concavities, and limits the use of the method in AR applications. In addition, only sparse features are used in Ruiz 2006 for pose estimation, with no error minimization step for increasing the accuracy of the pose estimated. 
         [0010]    Iterative optimization has been shown to be useful for tracking, as well as for refining given pose estimates. Fitzgibbon (hereinafter referred to as Fitzgibbon 2001) (Andrew W. Fitzgibbon, “Robust registration of 2D and 3D point sets”, In Proc. British Machine Vision Conference, volume II, pp. 411-420, 2001) proposed a 2D registration method for point sets based on the Levenberg-Marquardt nonlinear optimizer. As pointed out in Fitzgibbon 2001, direct nonlinear optimization on point sets can be easily extended to incorporate a robust estimator, such as a Huber kernel, which leads to more robust tracking. Such a method can also account for curves as sets of points, although the method makes no use of the connectivity information offered by such curves. 
         [0011]    A shape footprint, originally proposed by Lamdan et al. (hereinafter referred to as Lamdan 1988) (Lamdan, Y., Schwartz, J. T., and Wolfson, H. J., “Object Recognition by Affine Invariant Matching”, Computer Vision and Pattern Recognition, pp. 335-344, 1988.)), is a construction that can be used for calculating a signature for a shape. Shape footprints have been proposed for the recognition of flat and rigid objects undergoing affine transformations. 
       SUMMARY 
       [0012]    The present system provides a tool to estimate the relative pose of a generic object with respect to a camera view-point by processing 2D images from a monocular camera in real-time. The capability of solving the pose estimation problem relies on the robust detection and matching in consecutive image frames of significant visual features belonging to an object of interest. To accomplish this, the system incorporates 3D modeling of the object of interest. In one embodiment, the shape of interest may be approximated by a parametric surface such as a cylinder, sphere, cube, ellipsoid, or even complex non-parametric models. The system can restrain information retrieved at a 2D image level to estimate parameters about the pose. In operation, the accuracy of the 3D pose estimation of the object is a function of the degree of approximation of the selected model and the ability to select and track relevant features across consecutive image frames. 
         [0013]    These and further embodiments will be apparent from the detailed description and examples that follow. 
     
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
         [0014]    The present system is herein described, by way of example only, with reference to the accompanying drawings, wherein: 
           [0015]      FIG. 1  illustrates the three actors of an embodiment of the system. 
           [0016]      FIG. 2  is a flow diagram illustrating the operation of an embodiment of the system. 
           [0017]      FIG. 3  is a flow diagram illustrating the feature extraction of an embodiment of the system. 
           [0018]      FIG. 4  is a flow diagram illustrating pose estimation of an embodiment of the system. 
           [0019]      FIG. 5  is a block diagram of execution modules of an embodiment of the system. 
           [0020]      FIG. 6  is flow diagram of the operation of an embodiment of the system with feature repopulation. 
           [0021]      FIG. 7  is an example computer embodiment of the system. 
       
    
    
     DETAILED DESCRIPTION 
       [0022]    The present system is a method of 3D pose estimation of an object using 3D modelling of an approximation of the object. The approximation may be parametric or non-parametric as desired. 
         [0023]    The system is implemented by an acquisition system (e.g. camera) capturing an image (2D image plane) of an object in the 3D world. Each of these actors has its own coordinate system as illustrated in  FIG. 1 . Referring now to  FIG. 1  we see the camera  101  creating an image  102  from an object  104  in the real world  103 . The image  102  is a 2D image plane comprised of pixels and having a coordinate system having an origin 0,0 at the upper left in one embodiment, with rows  105  and columns  106  defining an image frame. 
         [0024]    The camera  101  is a video camera in one embodiment and operates in 3D space with 6DOF in the X c , Y c , Z c  coordinate system. The object of interest  104  is in the 3D real world and has a coordinate system in the X w , Y w , Z w  coordinate system. The camera and real world coordinate systems are related since they are both in 3D Cartesian space. 
         [0025]    The position and orientation of each of the actors is described by the displacement from the origin of the actor along the axes of its coordinate system. The origin may be taken anywhere in space. In one embodiment, the origin may be anchored to an easily identified location to optimize accuracy and ease of calculation. However, the origin can be defined anywhere without departing from the scope and spirit of the system. 
         [0026]    The object of interest  104  is the object in the real world whose 3D pose is to be estimated. The object of interest is shown in this example as a cylinder, but in reality the object of interest may be any natural shape. In the system the shape of interest is modeled in as another object so that it can be constrained and allow for more accurate pose estimation. 
         [0027]    The acquisition system in this embodiment is a camera  101  that can capture the 3D structure of the real world and transpose it onto a plane, referred to here as an image plane. 
         [0028]    The image plane  102  represents the results of the camera acquisition/projection process. Formally, the image actor  102  is strictly related to the camera. In fact, the camera is the medium by which the image can be obtained as projection of the real-world. According the camera model, several projection may be considered (e.g., perspective, orthographic, and the like). 
         [0029]    However, for the image plane  102 , the reference system is strictly 2D, allowing locating a point in a plane with 2 coordinates, commonly referred to as row and columns. Also, the displacements are given in one embodiment as picture elements, commonly known as pixels, the smallest addressable elements in a display device. 
         [0030]    The origin of the image system can be arbitrary taken anywhere on the image plane. A common choice is to center it on the top-left corner. 
         [0031]    Because the actors are in different coordinate systems, the system must perform geometric transformations to move from one coordinate system to the next. A 3D roto-translation of the real world and the camera system can be identified. This transformation describes the displacement of the camera origin with respect to the world origin. Working in 3D coordinate systems, the transformation can be formally defined as a concatenation of a 3×3 rotation matrix R, and a 3×1 translation vector T. The 3×4 matrix can be finally obtained through concatenation, i.e., [R\T]. 
         [0032]    In order to obtain the location in the image of some real-world point, a further geometrical transformation is required, to normalize the different distance measures (i.e., meters and pixel) of the 3 actors. This transformation is given by the so called camera matrix, also referred to as K This 3×3 matrix encompass all the technical camera information (i.e., intrinsic camera parameters) allowing the projection and transposition of a point in real-world system to the image plane. These parameters are in particular the camera focal length, the pixel dimension at sensor level, and the camera central point. In order to obtain this matrix a procedure referred to as calibration has to be carried out. Several techniques are available for the estimation of the camera matrix K and for purposes of one embodiment of the system, it can be assumed that K is given or known. 
         [0033]    The process by which a 3D point in the real-world is brought to the image plane through the camera is called projection, and will be referred to as P. In one embodiment, P can be defined as product between the matrix K, and the 31) roto-translation between the camera and the real-world. Mathematically, P=K[R\T]. 
         [0034]    In  FIG. 1 , a schematic reconstruction of the actor is reported, along with the geometric transformations required to go from one reference system to each of the others. 
         [0035]    Representing all the quantities in homogeneous coordinates, given a point X=(λ x , λ y , λ z , 1) in the real-world, its projection in the image plane x=(λ r , λ c , 1) can be found according the formula x=s PX, where the transformation is given up to a scaling factor s. 
         [0036]      FIG. 2  is a flow diagram illustrating the operation of the system in one embodiment. At step  201  the camera generates an image of the real world, including the object of interest, referred to as O. 
         [0037]    At step  202  the system extracts features from the image in order to generate a set of N image points x i  belonging to the object O. 
         [0038]    At step  203  the extracted features are used to generate a 3D projection of the image points x i  onto a reference model of the object O R . 
         [0039]    At step  204  the 3D projection is used to generate a pose estimation of the object of interest. Using the image points on the object O, a 3D model for approximating the image O, and the camera matrix K, we can estimate a pose of the object O as using matrix P=K[R\T]. 
         [0040]    Feature Extraction 
         [0041]    The feature extraction step  202  of  FIG. 2  phase allows the system to detect robust 2D patches of the object of interest. The patches should be such that it is possible to locate the object in different time instants (i.e. different image frames). In order to determine the presence in the scene of the object of interest, a classification process may be run. Alternatively, other procedures can be activated with the final aim of defining a bounding box containing the object in the image plane. 
         [0042]    In one embodiment, the feature extraction is a ‘blind’ procedure, where features are extracted regardless their physical meaning in the real-world and without a-priori knowledge of the object of interest. This allows the system to abstract from the nature of specific object, allowing the procedure to be applied to a wider range of scenarios. In other embodiments, where there is pre-knowledge of the specific object of interest, further information can be exploited to detect robust features (e.g., in face tracking it is possible to consider relevant facial features, such as eyes, nose, lips, and rely on their geometrical constraints to achieve a better detection and tracking). 
         [0043]    The system may be used with the a number of feature type and/or extraction techniques, including, but not limited to, corners, SIFT, SURF, or whatever other image feature that is possible to accurately locate object features in consecutive or non-consecutive image frames. 
         [0044]    The output of the feature extraction operation is a Bounding Box B of the object in the image plane along with an array of features or feature elements. The Bounding Box B is a rectangle in one embodiment with an origin point B p =(λ r , λ c ) defined in the image plane (with dimensions in rows and columns and expressed in pixels) being h and w, respectively. Thus the bounding box B is (B p , w, h). 
         [0045]    The features are defined as an array {f p } t , where p goes from 1 to FN and indexes the number of features found (e.g. FN), and t is a temporal index (i.e, which image frame is being analyzed). Each array element f p =(λ r , λ c , 1) locates in the image coordinates system each of the features found within the bounding box B. 
         [0046]      FIG. 3  is a flow diagram illustrating the feature extraction step  202  of  FIG. 2 . At step  301  the system receives another image frame at time t that includes the object of interest O. At step  302  the system defines the origin point B p  of the bounding box B. At step  303  the dimensions h and w of the bounding box B are determined. 
         [0047]    At step  304  the system finds features f within the bounding box B. At decision block  305  it is determined if there are more features to extract (e.g. has FN been reached). If so, the system returns to step  304 . If not, the system proceeds to step  306  and generates the feature array {f p } t . At step  307  the system outputs the boundary box and the feature array. 
         [0048]    Projection 
         [0049]    After the boundary box and feature set array have been generated, the system generates a projection of the feature set to locate the features onto the 3D reference model O R . In the example of  FIG. 1 , the model is cylindrical, but this is for purposes of example only. The reference model may be spherical, cylindrical, cubic, or some other parametric or non-parametric surface. 
         [0050]    The projection of the image feature points {f p } t , in the 3D real-world space is carried out relying on the adopted 3D reference model O R , and the bounding box B obtained in  FIG. 3 . 
         [0051]    In one embodiment, an orthographic projection is performed when bringing the 2D image feature points to the 3D object model. In this context, in order to simplify the calculations, the dimensions of the reference model can be adjusted so as to fit the bounding box B (e.g., if the model is a cylinder, and the bounding box is a rectangle with width and height of w and h, respectively, the cylinder height may be set to h, while its radius to w/2). Whenever a non-parametric model is considered, this procedure can be thought of as an over-quantization of the model surface. Also, the real-world reference system can be translated to some convenient location (i.e., the center of the considered 3D model). 
         [0052]    The output of this process is a set of samples {F p } t  in the real world coordinate system corresponding to the 2D image feature found at in the extraction step described above. It should be noted that other methods to project the feature array onto the reference model may be used without departing from the scope or spirit of the system. 
         [0053]      FIG. 4  is a flow diagram illustrating an embodiment of this process. At step  401  the system receives the bounding box B and the array {f p } t , from the feature extraction process. At step  402  the system compares the bounding box dimensions to the reference model O R . At decision block  403  it is determined if any adjustment or normalizing of the reference model dimensions is needed. If so, the dimensions are modified at step  404 . If not, the system proceeds to step  405  and does an orthographic projection of the feature points onto the model. 
         [0054]    At step  406  the system outputs the set of samples {F p } t  in the real world coordinate system. 
         [0055]    Pose Estimation 
         [0056]    Once the projection process has been carried out, a set of pairs of 2D feature samples in the image plane {f p }, and the corresponding 3D points {F p }, in the real-world coordinate system can be established (i.e. the correspondences are pairs of {f p , F p }) 
         [0057]    These correspondences, together with the camera matrix K, allow the effective estimation of the initial 3D object pose P with respect to the camera reference system. These sets of parameters are taken as reference for the successive processing steps and calculations. In fact several reference systems can be taken, also in non-Cartesian spaces (i.e., spherical), according to the algorithmic simplification that each system may offer in a particular case 
         [0058]    In this embodiment, the only parameter required for the pose estimation is the camera focal length. Concerning the core of the pose estimation algorithm, possible strategies include, but are not limited to, POSIT and SolvePnP. These are particular methods allowing the effective approximation of the object pose relying on iterative procedures, and approximate projection paradigms. The output of the pose estimation process is the pose at a point in time (based on the frame) P t =[R\T] 
         [0059]    In one embodiment, the system operates as shown in  FIG. 5 . The example of  FIG. 5  is after at least the first image frame has been processed. The camera  501  provides the next image frame from time t+1 to the Feature Set 2D Motion Estimation Block  502 . The feature array {f p } t  from the prior frame is also provided as input to the Feature Set 2D Motion Estimation Block  502 . 
         [0060]    The Feature Set 2D Motion Estimation Block  502  uses the image frame t+1 and the prior feature array {f p } t  to generate the new feature array {f p } t+1 . This operation is based on the assumption that sufficient of the previously identified features {f p } t  of frame t can be identified in new frame t+1. This will be true if inter-frame motion is small enough. The operation of Block  502  may employ any of several methods, alone or in combination, including optical flow, feature matching, and the like. The generated output is a set of new locations {f p } t+1  that represent the motion of each feature from the prior set {f p } t . The number of features that it is possible to locate across frames is q where q varies from 1 to Q and Q LTE FN. 
         [0061]    In operation, it should be noted that in some instances, not all of the features can be located across consecutive frames, regardless their robustness (e.g., in case of occlusion with other objects, or self-occlusions, and the like). Although a percentage of starting feature set {f p } t  may not be tracked in the frame t+1, a number of pairs {f q , F q } t+1  can be found. 
         [0062]    It is important to note that {F p } t ={F p } t+1  since the real-world reference system is moving together with the model O. This implies that {F q } t+1  is a subset of {F p } t+1 . 
         [0063]    The output {f p } t+i , along with the real world sample points {F p } t  from the previous image frame, are provided to the Pose Estimation Block  503 . The Pose Estimation Block  503  uses this data to generate the new pose P t+1  for the latest frame. Typically, the Pose Estimation Block  503  will use {F q } t+1  and {f q } t  as the determined features and points may be less than the number from the prior frame. 
         [0064]    The new pose P t+1  is provided to the Feature Set Repopulation Block  504 , along with the current image frame t+1. The Feature Set Repopulation Block  504  generates the new correspondence pairs {f p , F p } t+1  that can be used for the next operation when the next image frame is acquired. 
         [0065]    The purpose of the Feature Set Repopulation Block  504  is to maintain a consistent number of pairs of feature points across image frames t and t+1. To that end, the Feature Set Repopulation Block  504  extracts a new set of image features {nif n } t+1  where n varies from 1 to FN-Q. These new features are extracted from the current frame to maintain a consistent number with the prior frame. 
         [0066]    The pose P(t+1) of the object O has already been estimated in the t+1 frame. Thus, the projection step can accurately locate the new feature set in the 3D model, obtaining {F n } (t+1) . This allows the problem of lost features to be overcome and allows the system to work with the same number of features FN. 
         [0067]    The ultimate output of the system is an array of pairs {f p , F p } t+1  where the feature set {f p }={f q } t+1  {nif n }( t+1) , while F p ={F q } t +{F n } (t+1) . In other words, the feature set at time t+1 is a composition of the features found at time t that also are found in time t+1 and a new set of features that has been retrieved in the frame at time t+1. 
         [0068]    In the same manner, the 3D points in the real world are given by the projections at time t of {f q } (t+1)  (unchanged since they refer to the real world reference system) and the projections in the real world of the feature set {nif n } (t+1)  that are obtained from the feature extraction step while considering the pose P t+1  of the new frame. 
         [0069]    At this point, the number of available correspondence pairs {f p , F p } where p is from 1 to FN is available. 
         [0070]      FIG. 6  is a flow diagram illustrating the operation of an embodiment of the system. At step  601  the system receives the new image frame. At step  602 , the system receives the next image frame. At step  602  the system uses the prior array data and other parameters to define the boundary box and to extract features from the new frame at step  603 . At step  604  the system uses the extracted features, boundary box, and prior real world sample points to perform a pose estimation of the model. 
         [0071]    At step  605  the system assesses the extracted features and at decision block  606  it is determined if the extracted features are less than FN. If so, the system repopulates the features accordingly at step  607 . After step  607 , or if there are enough features at step  606 , the system updates the parameter that will be used for the next incoming image frame and returns at step  609  to receive the next image frame. 
         [0072]    Embodiment of Computer Execution Environment (Hardware) 
         [0073]    An embodiment of the system can be implemented as computer software in the form of computer readable program code executed in a general purpose computing environment such as environment  700  illustrated in  FIG. 7 , or in the form of bytecode class files executable within a Java™ run time environment running in such an environment, or in the form of bytecodes running on a processor (or devices enabled to process bytecodes) existing in a distributed environment (e.g., one or more processors on a network). A keyboard  710  and mouse  711  are coupled to a system bus  718 . The keyboard and mouse are for introducing user input to the computer system and communicating that user input to central processing unit (CPU  713 . Other suitable input devices may be used in addition to, or in place of, the mouse  711  and keyboard  710 . I/O (input/output) unit  719  coupled to bi-directional system bus  718  represents such I/O elements as a printer, A/V (audio/video) I/O, etc. 
         [0074]    Computer  701  may be a laptop, desktop, tablet, smart-phone, or other processing device and may include a communication interface  720  coupled to bus  718 . Communication interface  720  provides a two-way data communication coupling via a network link  721  to a local network  722 . For example, if communication interface  720  is an integrated services digital network (ISDN) card or a modem, communication interface  720  provides a data communication connection to the corresponding type of telephone line, which comprises part of network link  721 . If communication interface  720  is a local area network (LAN) card, communication interface  720  provides a data communication connection via network link  721  to a compatible LAN. Wireless links are also possible. In any such implementation, communication interface  720  sends and receives electrical, electromagnetic or optical signals which carry digital data streams representing various types of information. 
         [0075]    Network link  721  typically provides data communication through one or more networks to other data devices. For example, network link  721  may provide a connection through local network  722  to local server computer  723  or to data equipment operated by ISP  724 . ISP  724  in turn provides data communication services through the world wide packet data communication network now commonly referred to as the “Internet”  727  Local network  722  and Internet  727  both use electrical, electromagnetic or optical signals which carry digital data streams. The signals through the various networks and the signals on network link  721  and through communication interface  720 , which carry the digital data to and from computer  700 , are exemplary forms of carrier waves transporting the information. 
         [0076]    Processor  713  may reside wholly on client computer  701  or wholly on server  727  or processor  713  may have its computational power distributed between computer  701  and server  727 . Server  727  symbolically is represented in  FIG. 7  as one unit, but server  727  can also be distributed between multiple “tiers”. In one embodiment, server  727  comprises a middle and back tier where application logic executes in the middle tier and persistent data is obtained in the back tier. In the case where processor  713  resides wholly on server  727 , the results of the computations performed by processor  713  are transmitted to computer  701  via Internet  727 , Internet Service Provider (ISP)  724 , local network  722  and communication interface  720 . In this way, computer  701  is able to display the results of the computation to a user in the form of output. 
         [0077]    Computer  701  includes a video memory  714 , main memory  715  and mass storage  712 , all coupled to bi-directional system bus  718  along with keyboard  710 , mouse  711  and processor  713 . 
         [0078]    As with processor  713 , in various computing environments, main memory  715  and mass storage  712 , can reside wholly on server  727  or computer  701 , or they may be distributed between the two. Examples of systems where processor  713 , main memory  715 , and mass storage  712  are distributed between computer  701  and server  727  include thin-client computing architectures and other personal digital assistants, Internet ready cellular phones and other Internet computing devices, and in platform independent computing environments, 
         [0079]    The mass storage  712  may include both fixed and removable media, such as magnetic, optical or magnetic optical storage systems or any other available mass storage technology. The mass storage may be implemented as a RAID array or any other suitable storage means. Bus  718  may contain, for example, thirty-two address lines for addressing video memory  714  or main memory  715 . The system bus  718  also includes, for example, a 32-bit data bus for transferring data between and among the components, such as processor  713 , main memory  715 , video memory  714  and mass storage  712 . Alternatively, multiplex data/address lines may be used instead of separate data and address lines. 
         [0080]    In one embodiment of the invention, the processor  713  is a microprocessor such as manufactured by Intel, AMD, Sun, etc. However, any other suitable microprocessor or microcomputer may be utilized, including a cloud computing solution. Main memory  715  is comprised of dynamic random access memory (DRAM). Video memory  714  is a dual-ported video random access memory. One port of the video memory  714  is coupled to video amplifier  719 . The video amplifier  719  is used to drive the cathode ray tube (CRT) raster monitor  717 . Video amplifier  719  is well known in the art and may be implemented by any suitable apparatus. This circuitry converts pixel data stored in video memory  714  to a raster signal suitable for use by monitor  717 . Monitor  717  is a type of monitor suitable for displaying graphic images. 
         [0081]    Computer  701  can send messages and receive data, including program code, through the network(s), network link  721 , and communication interface  720 . In the Internet example, remote server computer  727  might transmit a requested code for an application program through Internet  727 , ISP  724 , local network  722  and communication interface  720 . The received code maybe executed by processor  713  as it is received, and/or stored in mass storage  712 , or other non-volatile storage for later execution. The storage may be local or cloud storage. In this manner, computer  700  may obtain application code in the form of a carrier wave. Alternatively, remote server computer  727  may execute applications using processor  713 , and utilize mass storage  712 , and/or video memory  715 . The results of the execution at server  727  are then transmitted through Internet  727 , ISP  724 , local network  722  and communication interface  720 . In this example, computer  701  performs only input and output functions. 
         [0082]    Application code may be embodied in any form of computer program product. A computer program product comprises a medium configured to store or transport computer readable code, or in which computer readable code may be embedded. Some examples of computer program products are CD-ROM disks, ROM cards, floppy disks, magnetic tapes, computer hard drives, servers on a network, and carrier waves. 
         [0083]    The computer systems described above are for purposes of example only. In other embodiments, the system may be implemented on any suitable computing environment including personal computing devices, smart-phones, pad computers, and the like. An embodiment of the invention may be implemented in any type of computer system or programming or processing environment. 
         [0084]    While the system has been described with respect to a limited number of embodiments, it will be appreciated that many variations, modifications, and other applications of the system may be made.