METHOD AND APPARATUS FOR CAPTURING 360 DEGREE VIEWING IMAGES USING SPHERICAL CAMERA AND MOBILE PHONE

One aspect of the present invention discloses an image capturing system able to capturing 3-dimensional 360 degree viewing (“3D/360”) images using a smart phone and a ball-shaped spherical camera. The smart phone includes a digital image processor (“DIP”) capable of processing audio and video (“AV”) information. The camera includes a set of surface-mount lenses evenly distributed over the surface of spherical shaped camera. The camera further includes lens motion controllers and image sensors which are used to sense images captured by the lenses. The lens motion controllers are configured to independently control lens position or orientation for each surface-mount lens. To process and display 3D/360 images, the camera transmits 3D/360 images related information to the smart phone via a communication network, such as Ethernet cable, wireless channel, cellular channel, and/or Bluetooth® wireless network.

FIELD

The exemplary embodiment(s) of the present invention relates to telecommunications network. More specifically, the exemplary embodiment(s) of the present invention relates to digital image processing.

BACKGROUND

With increasing network capacity capable of handling voluminous multimedia information in a high-speed communications network, high quality and high resolution images and/or videos for audio video (“AV”) data are in high demand. A conventional camera system capable of capturing and processing quality 3 dimensional (“3D”) and/or 360 degree viewable pictures/frames typically requires sophisticated optical lens or lenses as well as high-speed AV image computing resource. A drawback associated with a conventional high quality camera capable of taking 3D or 360 degree viewable pictures is that it is bulky, heavy, and difficult to operate. For example, a typical 3D/360 camera also known as an omnidirectional camera is relatively large and bulky that is difficult to integrate into a mobile phone.

Another common problem associated with a conventional 3D/360 camera is that it requires unique lenses and special imaging processing hardware to manipulate a continuous panoramic view. For example, captured 2-dimensional (“2D”) image does not provide the image data needed to produce a 3D model or image. In addition, simultaneous zooming and focusing each camera generally consume large amount of hardware resource.

SUMMARY

The following summary illustrates a simplified version(s) of one or more aspects of present invention. The purpose of this summary is to present some concepts in a simplified description as more detailed description that will be presented later.

One embodiment of present invention discloses an image capturing system able to capturing 3-dimensional 360 degree viewing (“3D/360”) images using a smart phone and a ball-shaped spherical camera. The smart phone includes a digital image processor (“DIP”) capable of processing audio and video (“AV”) information. The camera includes a set of surface-mount lenses evenly distributed over the surface of spherical shaped camera. The camera further includes lens motion controllers and image sensors which are used to sense images captured by the lenses. The lens motion controllers are configured to independently control lens position or orientation for each surface-mount lens. To process and display 3D/360 images, the camera transmits 3D/360 images related information to the smart phone via a communication network, such as Ethernet cable, wireless channel, cellular channel, and/or Bluetooth® wireless network.

DETAILED DESCRIPTION

Embodiments of the present invention are described herein in the context of methods and/or apparatus for providing 3D/360 degrees camera (“3DC”) system capable of capturing 3D/360 images using a spherical camera and a mobile phone.

The purpose of the following detailed description is to provide an understanding of one or more embodiments of the present invention. Those of ordinary skills in the art will realize that the following detailed description is illustrative only and is not intended to be in any way limiting. Other embodiments will readily suggest themselves to such skilled persons having the benefit of this disclosure and/or description.

In the interest of clarity, not all of the routine features of the implementations described herein are shown and described. It will, of course, be understood that in the development of any such actual implementation, numerous implementation-specific decisions may be made in order to achieve the developer's specific goals, such as compliance with application- and business-related constraints, and that these specific goals will vary from one implementation to another and from one developer to another. Moreover, it will be understood that such a development effort might be complex and time-consuming, but would nevertheless be a routine undertaking of engineering for those of ordinary skills in the art having the benefit of embodiment(s) of this disclosure.

Various embodiments of the present invention illustrated in the drawings may not be drawn to scale. Rather, the dimensions of the various features may be expanded or reduced for clarity. In addition, some of the drawings may be simplified for clarity. Thus, the drawings may not depict all of the components of a given apparatus (e.g., device) or method. The same reference indicators will be used throughout the drawings and the following detailed description to refer to the same or like parts.

The term “system” or “device” is used generically herein to describe any number of components, elements, sub-systems, devices, packet switch elements, packet switches, access switches, routers, networks, modems, base stations, eNB (eNodeB), computer and/or communication devices or mechanisms, or combinations of components thereof. The term “computer” includes a processor, memory, and buses capable of executing instruction wherein the computer refers to one or a cluster of computers, personal computers, workstations, mainframes, or combinations of computers thereof.

IP communication network, IP network, or communication network means any type of network having an access network that is able to transmit data in a form of packets or cells, such as ATM (Asynchronous Transfer Mode) type, on a transport medium, for example, the TCP/IP or UDP/IP type. ATM cells are the result of decomposition (or segmentation) of packets of data, IP type, and those packets (here IP packets) comprise an IP header, a header specific to the transport medium (for example UDP or TCP) and payload data. The IP network may also include a satellite network, a DVB-RCS (Digital Video Broadcasting-Return Channel System) network, providing Internet access via satellite, or an SDMB (Satellite Digital Multimedia Broadcast) network, a terrestrial network, a cable (xDSL) network or a mobile or cellular network (GPRS/EDGE, or UMTS (where applicable of the MBMS (Multimedia Broadcast/Multicast Services) type, or the evolution of the UMTS known as LTE (Long Term Evolution), or DVB-H (Digital Video Broadcasting-Handhelds)), or a hybrid (satellite and terrestrial) network.

One embodiment of the presently claimed invention discloses an image capturing system able to capturing 3D/360 images using a smart phone and a ball-shaped spherical camera. In one example, the image capturing system can also be referred to as a 3D/360 degree camera (“3DC”) system. The smart phone includes a digital image processor (“DIP”) capable of processing audio and video (“AV”) information. The camera includes a set of surface-mount lenses evenly distributed over the surface of spherical shaped camera. The camera further includes lens motion controllers and image sensors which are used to sense images captured by the lenses. The lens motion controllers are configured to independently control lens position or orientation for each surface-mount lens. To process and display 3D/360 images, the camera transmits 3D/360 images related information to the smart phone via a communication network, such as Ethernet cable, wireless channel, cellular channel, and/or Bluetooth® wireless network.

FIG. 1is a block diagram illustrating a communication network containing an image capturing system capable of capturing 3D/360 images using a spherical camera in accordance with one embodiment of the present invention. Diagram100illustrates 3D/360 image capturing component106, communication network102, switching network104, Internet150, and portable electric devices114-120. In one aspect, network102can be wide area network (“WAN”), metropolitan area network (“MAN”), local area network (“LAN”), satellite/terrestrial network, or a combination of WAN, MAN, and LAN. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or networks) were added to or removed from diagram100.

Network102includes multiple network nodes, not shown inFIG. 1, wherein each node may include mobility management entity (“MME”), radio network controller (“RNC”), serving gateway (“S-GW”), packet data network gateway (“P-GW”), or HomeAgent to provide various network functions. Network102is coupled to Internet150, my-status server108, base station112, and switching network104. Server108, in one embodiment, includes my-status management or my-status106which can be software, hardware, or combination of software and hardware component.

Switching network104, which can be referred to as packet core network, includes cell sites122-126capable of providing radio access communication, such as 3G (3rdgeneration), 4G, or 5G cellular networks. Switching network104, in one example, includes IP and/or Multi Protocol Label Switching (“MPLS”) based network capable of operating at a layer of Open Systems Interconnection Basic Reference Model (“OSI model”) for information transfer between clients and network servers. In one embodiment, switching network104is logically coupling multiple PEDs114-120across a geographic area via cellular networks. It should be noted that the geographic area may refer to a campus, city, metropolitan area, country, continent, or the like.

Base station112, also known as cell site, node B, or eNodeB, includes a radio tower capable of coupling to various user equipments (“UEs”), PEDs, or spherical camera166. The term UEs and PEDs can be referred to similar portable devices and they can be used interchangeably. For example, UEs or PEDs can be cellular phone114, handheld device118, iPhone®116, tablets and/or iPad® 120 via wireless communications. Handheld device118can be a smart phone, such as iPhone®, BlackBerry®, Android®, Samsung Galaxy®, and so on. Base station112, in one example, facilitates network communication between mobile devices such as portable handheld device114-120via wired and wireless communications networks. It should be noted that base station112may include additional radio towers as well as other land switching circuitry.

iPhone®116, in one embodiment, includes a spherical camera160capable of capturing 3D/360 panoramic pictures and/or videos. Spherical camera160as shown in an exploded view162includes multiple lenses mounted on the surface of spherical camera160. In one aspect, lenses are evenly distributed on the surface of spherical camera160for capturing a 360-degree viewing image. Spherical camera160, in one embodiment, is coupled to iPhone®116via a wire, Bluetooth®, WiFi, cellular network, local wireless network, Ethernet, and the like.

Internet150is a computing network using Transmission Control Protocol/Internet Protocol (“TCP/IP”) to provide linkage between geographically separated devices for communication. Internet150, in one example, couples to supplier server138and satellite network130via satellite receiver132. Satellite network130, in one example, can provide many functions as wireless communication as well as global positioning system (“GPS”). For example, my-status106can receive GPS information from satellite network130via Internet150, network102, and switching network104.

Independent spherical camera (“ISC”) or spherical camera166, in one embodiment, includes multiple lenses170and a transceiver168wherein transceiver168can be a wireless transceiver or land based transceiver. A function of ISC166is to capture 3D/360 image or images using one or more lenses170. Upon capturing 3D/360 image information, it is transmitted to its destination via a communication network such as network102. 3D/360 component106, in one aspect, is coupled to a network such as network102to provide 3D/360 transmission services including distribution of network software for handling 3D/360 data. In one embodiment, 3D/360 component106resides at a server and provides 3D/360 image application (“App”) that can be installed or downloaded to portable phones.

During an exemplary operation, a user of portable device119downloads the App for 3D/360 component106via network102. Upon installation of the App, device119, in one embodiment, is able to communicate with ISC166and can request specific 3D/360 images or image information from ISC166. Upon capturing the requested 3D/360 images, ISC166transmits imaging data to portable device119via network102.

An advantage of employing spherical camera is to provide 3D/360 images using smart phone's computing capabilities.

FIG. 2is a block diagram200illustrating a spherical camera166coupled to a portable device116capable of capturing 3D/360 images using multiple surface-mount lenses in accordance with one embodiment of the present invention. Portable device166can be an iPhone®, smart phone, iPad®, Samsung Galaxy®, and/or laptop which is capable of connecting to spherical camera166via direct wireless channel236, cellular channels232-234, land line230, and the like. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or networks) were added to or removed from diagram200.

Diagram200illustrates spherical camera166with a portion of its spherical shaped camera202cut-open along line204. Spherical camera166, in one embodiment, includes multiple lenses210wherein each of multiple lenses210is supported by a lens motion controller212and an image sensor214. To capture 3D/360 pictures and/or videos, a central processing unit (“CPU”)206is employed in spherical camera166. While CPU206is coupled to every image sensor such as image sensor214via connection216, CPU206is also coupled to a transceiver208. Transceiver208, in one example, is a combination of transmitter and receiver capable of performing transmission and receiving functions over a communications network. In one aspect, transceiver208can be integrated into CPU206as a single chip.

Transceiver208, in one aspect, includes an antenna218and an Ethernet port220wherein port220can be used to receive an Ethernet plug222such as an RJ45 plug for coupling spherical camera166to a land line230via a local area network (“LAN”)224. Antenna218, in one example, can be used to communicate with wireless network(s) such as cellular network, Bluetooth, and/or WiFi network. It should be noted that WiFi can be a local area wireless computing network.

Diagram200illustrates a 3D/360 degree camera (“3DC”) system containing a smart phone116such as an iPhone®, spherical camera166, and communication networks. Smart phone116includes a digital image processor (“DIP”) configured to process audio and video information. Spherical camera166, in one embodiment, contains a set of surface-mount lenses210that are evenly distributed over the surface of spherical camera166. Spherical camera166, in one example, is configured to capture 3D/360 images. Spherical camera166further includes a group of image sensors such as sensor212and lens motion controllers such as controller214.

The image sensors are coupled to surface-mount lenses210and are capable of sensing images, such as 3D, 360 degrees, or 3D/360 images or image information. The lens motion controllers are coupled to surface-mount lenses210and are configured to independently control lens position for each of surface-mount lenses210. In one embodiment, spherical camera166has transceiver208able to transmit 3D/360 images or 3D/360 image information to smart phone116via a communication network. The communication network can be an Ethernet cable, wireless channel, cellular channel, Bluetooth® wireless network, or a combination of Ethernet, wireless channel, cellular channel, and Bluetooth® wireless network.

A function of 3DC system is able to capture 3D/360 images using a ball-shaped camera such as spherical camera166and a mobile phone such as iPhone® 116. The 3DC system which can also be referred to as image capturing system couples ball-shaped camera166to mobile phone116using one or more communication networks. In one example, the computing capability of mobile116is used to process 3D/360 images captured by the ball-shaped camera. With shifting a portion of image processing task from the camera to the mobile phone(s), the camera's functions and/or hardware is reduced whereby the overall size of spherical shaped camera is reduced. For instance, spherical camera may be fabricated with small and compact physical dimension(s) or attributes that can be easily integrated into or attached to a mobile phone.

In one example, a spherical, also known as 3D/360 degrees, camera is structured with a spherical round shape hosting a set of surface-mount lenses210. Lenses210, in one example, are covered or distributed evenly over the surface of spherical camera such as camera166. Each lens of the surface-mount lenses210is attached to a motion controller such as controller214and an image sensor such as sensor212. Motion controller214is connected to CPU206wherein CPU206controls movement of each lens to produce required zooming or focusing effect. An advantage of using surface-mount lenses is that the mounting method optimizes and/or minimizes overall size or dimension of the ball-shaped 3D/360 camera.

During an operation, at least a portion of multiple surface-mount lenses are instructed to take a picture with approximately simultaneous exposure. Multiple images associated with the picture are captured by the portion of multiple surface-mount lenses and a spherical distribution of viewing picture is generated using the multiple captured images. These images are subsequently processed using an image processor to produce one combined spherical image.

The 3DC system, in one embodiment, is able to control each surface-mount lens to determine whether it should be activated to take a picture. The 3DC system is configured to control zooming effect and/or orientation of each surface-mount lens for capturing a wider or narrower image(s). For example, lenses which are distributed around the middle of the sphere can be instructed to capture an image while other lenses are inactivated to obtain a 360-degree panoramic image but not a 3D image. An advantage of using a 3DC system is that it is able to capture a full 360-degree panoramic view. Another advantage of using a 3DC system is that it is capable of capturing full 360 degrees panoramic view in multiple directions spanning across the sphere shape to produce a full 360 degrees, 3D image of the captured scene.

FIG. 3is a logic block diagram300illustrating a process of 3DC system able to handle various 3D/360 images captured by a set of lenses mounted on the surface of spherical camera in accordance with one embodiment of the present invention. Diagram300includes a mobile116, a spherical camera301, and network102wherein spherical camera301is coupled to mobile116via network connections such as wireless channel354or network102. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or networks) were added to or removed from diagram300.

Spherical camera301, in one embodiment, includes a camera CPU308, transceiver320, and a group of lens components302-306. Each of lens components302-306includes a lens310, image sensor312, and motion controller314. CPU308is configured to control each lens component based on instructions received from mobile116via transceiver320. CPU308is also able to activate or deactivate selected lens components based on instructions and/or commands sent from mobile116. Each motion controller314is controlled by CPU308to move or orient lens310for zooming or focusing effect. CPU308provides orientation and/or moving instruction to each lens component such as lens component302based on commands from mobile116.

Transceiver320is used to communicate with transceiver330located at mobile116via wired or wireless communications networks. For example, transceiver320can communicate with transceiver330via WiFi network or Bluetooth network via channel354. Alternatively, transceiver320may reach transceiver330via cellular network via channels356-358. Also, transceiver320can couple to transceiver322via network102via channels350-352and radio towers112. It should be noted that a wire or cable connection is also possible between transceivers320-330.

Mobile116, which can be a portable processing device and smart phone, includes transceiver330, image processing332, and image display334. In one embodiment, mobile116is configured to include a DIP capable of performing image processing332. In one aspect, DIP of mobile116is capable of instructing spherical camera301that the type of image(s) should be taken. Image display334, for example, is able to display real-time images or videos captured by spherical camera301which could be placed miles away.

Diagram300illustrates a 3DC or image capturing system containing a portable processing device116and a camera301. Portable processing device116includes a DIP configured to process AV information. In one example, portable processing device116is an iPhone®, iPad®, Samsung Galaxy®, or smartphone. In one aspect, portable processing device116includes an image displaying mechanism capable of displaying 3D/360 images and/or video.

Camera301, in one embodiment, is structured with a spherical shape, and contains CPU308, which is logically coupled to the DIP of portable processing device116. Camera301is able to capture 3D/360 images. Camera301, also known as spherical camera, includes a ball-shaped structure, lenses310, and lens motion controllers314. The ball-shape, round-shape, or elongated shape camera is configured to house CPU308. Lenses310are physically mounted on the spherical surface of camera301capable of capturing images. Lens motion controllers314are coupled to lenses310and are configured to independently control the movement of each lens310that is amounted on the spherical surface of camera301.

Camera301further includes various image sensors312coupled to lenses310wherein image sensors312are able to digitize various images captured by lenses310. Image sensors312, in one embodiment, generate or digitize sensed image information and subsequently forward sensed image information to CPU308. In one example, image sensors312are charge-coupled devices (“CCDs”), also known as CCD image sensors. A function of a CCD is to visualize pixels collected in light of photons.

To communicate with portable processing device116, camera301employs a transceiver which is able to transmit the sensed images to portable processing device116via a communication channel. The transceiver, in one embodiment, includes an antenna capable of communicating with regional base stations. The communication channel, in one example, can be an Ethernet cable, wireless channel, cellular channel, Bluetooth® wireless network, and the like.

In one embodiment, a portion of lenses310is categorized as primary lenses while the remaining portion of lenses310are considered as surrounding lenses310. The lenses are designated as primary lenses are primarily based on their physical locations on the surface of spherical camera. For example, a spherical camera may contain 18 lenses wherein six (6) lenses are primary lenses while twelve (12) lenses are surrounding lenses.

An advantage of using a 3DC system with a spherical camera is that it allows control of each lens to define separate zooming effects and/or other types of control allowed by digital cameras. Note that the system allows taking simultaneous exposures of all lenses as well as separate or independent exposures to produce different types of images. Upon receiving multiple images for the same exposures, various images may be combined and manipulated while overlapping portions of images are separated and removed.

The 3DC system allows lenses to be mounted separately from the control mechanism and image sensor for reducing overall dimension of camera301. Camera301, in one example, is controlled by a mobile phone such as portable processing device116. The camera, in one aspect, is mounted on top of a mobile phone. Alternatively, the camera can be remotely located and is connected to the mobile phone via a wireless signal. Note that single pixel manipulation within each image can be implemented. Also, pixel filter may be added to refine the captured images.

FIG. 4is a physical diagram400illustrating an exemplary spherical camera configured to assist a portable device to capture 3D/360 images in accordance with one embodiment of the present invention. Diagram400illustrates a spherical shaped camera capable of housing multiple lenses310which are evenly distributed on the surface of camera in such a way that captures multiple zones of surroundings.

FIG. 5is a physical block diagram500illustrating a view of spherical camera showing distribution of lenses with overlapping views in accordance with one embodiment of the present invention. Diagram500illustrates a distribution of lenses on the spherical surface of the camera to capture different zones including surrounding scene. It should be noted that a set of lenses distributed on the perimeter of the spherical surface can be repeated up and down on the spherical surface of camera. The distribution of lenses can generate overlaps between different images captured by different lenses. In one embodiment, an imaging processing to identify and remove overlapping portions of 3D/360 image will be described inFIGS. 8-9.

Each lens, in one aspect, can be controlled to define the exposure time and zooming effect separately or jointly with other lenses. Such control of each lens allows a user to define preferred image content. Also, such control of each lens allows a user to capture a specific image with one or more lenses. A user, in one example, uses 3DC system to handle or combine images captured in multiple ways with different effects by various lenses.

The camera or spherical camera, in one example, is controlled by a mobile phone or other mobile devices which communicate with the camera via a Bluetooth, WiFi, or other means of wired or wireless communication networks. An advantage of using a 3DC system with a spherical camera is that it enables a user to control lenses and lenses orientations to take user defined or preferred 3D, 360-degree, or 3D/360 images or videos.

FIG. 6is a block diagram600illustrating a structure of spherical camera602used in conjunction with a smart phone (not shown) to capture 3D/360 images in accordance with one embodiment of the present invention. Diagram600illustrates a spherical shaped camera602hosting multiple lenses608on surface616of the camera. Camera602is configured to house a digital signal processor612and a lenses controller614. In one embodiment, camera602is structured in a compact size using semiconductor fabrication process and/or MEMS (microelectromechanical systems) fabrication process to assemble or process lenses608with image sensors610and lens control606. It should be noted that the underlying concept of the exemplary embodiment(s) of the present invention would not change if one or more blocks (or components) were added to or removed from diagram600.

To reduce overall size of a camera, various surface-mount lenses are used. Micro-machinery assembly such as MEMS assembly process may be used to manufacture spherical camera during an assembly of mobile phones. Depending on the applications, camera sizes can vary with different applications for multiple configurations. Lens control606, which may be manufactured and attached with lens608, includes lens movement part(s) which can be used to move lens608backwards and forward for zooming. Lenses608can also be moved up or down for providing adjustment(s) to picture or image(s) to be taken without movement of camera602. Image sensor610, in one embodiment, is an image capturing chip (or die or circuitry) which is connected to the image processor for imaging processing. In one embodiment, different types of lenses with different types of optical quality are used for enhancing image capturing. In one embodiment, multiple lenses are used to identify or measure distance. Alternatively, multiple lenses are used to identify or measure speed of a moving object such as a car or bird.

An advantage of having a compact camera is that it can be integrated into a smart phone.

FIG. 7is a diagram700illustrating an operation of capturing and processing captured images using spherical camera and a mobile phone in accordance with one embodiment of the present invention. During an operation, a user opens an application (“App”) on his or her mobile device116and subsequently could see at least a part of intended picture or video to be taken on the display as indicated by numeral1. The user may choose either the picture or video. The App, in one example, is previously installed on mobile device116. After the app sends a capture or record command to camera166via Bluetooth or GSM data link as indicated by numeral2, camera166takes a 3D/360 panoramic picture or video702as indicated by numeral4. Upon capturing the image information related to 3D, 360-degree, or 3D/360 image(s), the image information is transmitted back to mobile device116as indicated by numeral3. Note that camera166can also transmit the image information to other different mobile devices concurrently. Upon receiving the image information by mobile device116, the image information is processed by mobile device116to generate displayable 3D/360 image(s) and/or videos via the display of mobile device116. It should be noted that the image information, in one aspect, is not displayable until the information is processed by mobile device116.

To capture a full 360 degrees and/or 3D viewable image(s) with all directions, the distribution of lenses over the surface of camera and the image processing algorithm are important attributes to generate high quality/high resolution 3D/360 images or videos. Depending on the number of lenses, the image processing algorithm is used to calibrate and/or configure distribution of lenses to produce a complete or user defined 3D/360 picture or image.

FIG. 8is a logic diagram illustrating an image processing procedure capable of processing 3D/360 images captured by surface-mount lenses in accordance with one embodiment of the present invention. In one embodiment, image processing algorithm is performed as follows:The coordinate of lens location within the spherical shape is:

L(x,y,z)where x,y,z are coordinates of lens location in x-axis, y-axis, and z-axis. Image for alens is identified as I(L), whereby,

Lens Location=L(x,y,z)Image for this location is I(L)Pixels map of an image is vector multiplication of actual pixel location within thecaptured image as define in XY coordinate in the image and the corresponding lenslocation can be identified as:

The above equation gives a definition of every pixel location in terms of the lens location and the pixel location within an image frame. The algorithm then removes over lapping pixels by comparing this position of the pixel with similar pixel location from another image. The algorithm keeps one of the pixels in the final combined image. An exemplary process of implementing the algorithm is illustrated in logic diagram800.

At block802, the process calculates pixel location per each lens viewed or took the image. After comparing pixel locations associated with different images taken by different lenses at block804, the process exams whether the pixel locations are the same at block806. The process proceeds to block814to combine the pixels if the pixel locations are not the same at block806. Otherwise, the process proceeds to block808to exam whether the lens view is primary. If the lens view is not primary, the process proceeds to block812to remove the pixel. If the lens view is primary at block808, the process proceeds to block810to keep the pixel. After combining the pixels at block814, the process proceeds to block816to determine whether the pixel filter is required. If the pixel filter is required, the process proceeds to block820to apply the filter. Otherwise, the process proceeds to block818to indicate that the process is done or terminated.

It should be noted that the algorithm of imaging process starts with comparing location of the pixels taken by one of the lenses with the surrounding lenses mage pixels.

FIG. 9is a block diagram900illustrating primary view902captured by primary lens and surrounding lens views904captured by the surrounding lenses in accordance with one embodiment of the present invention. Diagram900illustrates one primary lens view900and six (6) surrounding lens views904. The areas of overlapping between primary view900and surrounding lens views are overlapping areas906. The algorithm starts with comparing location of the pixels taken by one of the lenses with the surrounding lenses mage pixels. The comparison of pixels is performed by a looping process which reiterates its process (or comparison) until all pixels are exhausted or depleted.

Same comparison is done for lenses. In every comparison, one lens is designated as primary and the other surrounding lenses are defined as secondary. If two pixels locations are similar when the comparison is done, the pixel from the primary is kept and the pixel from the secondary is removed.

FIG. 10is a physical block diagram illustrating a spherical camera1000showing distribution of primary and surrounding lenses in accordance with one embodiment of the present invention. In one example, camera1000has six (6) primary lenses and twelve (12) surrounding lenses. Depending on the applications, more or less primary or surrounding lenses may be configured and used.

Each of lens images after the comparison is kept and designated with its unique lens number. Once all comparisons are done, final combined image or spherical image the user sees is composed from all of these images or pixels. The final image can be described as the sum of all pixels as:

Final Pixel(x,y)=∫Pix(I)dx(I) where the limits of the integral is zero to total number of lenses used.

While the captured pixels are being calculated and compared pixels between multiple captured images, pixels can also be manipulated at the same time. For example, a filter may be added to the pixels. Another feature of controlling each lens is that it allows the user to control camera functions, such as shutting a single lens, grouping a cluster of lenses, forcing all lenses to capture images, zooming, and the like. The control within each lens is done by a program running on the mobile phone which is capable of displaying a map of each lens used in the camera. The user can define which lens to be used and which lens to be deactivated.

Once the App is selected then the program communicates with the control to define these settings and takes the picture. For example, when a picture is taken and processed according to the described algorithm, it is then returned back or communicated back to the phone where it can be viewed via viewer which is part of the mobile application. An advantage of using 3DC system is that it provides a true 3 dimensional and 360 degrees image capture using a configurable assembly which is fully controlled by a user and not fixed in its own assembly.

The image processing algorithm can produce both still images and video output. In one embodiment, the processes images frame by frame. By changing the processing power of the image processor, the quality of images is produced to be used with different type of mobile phones or as standalone camera.

FIG. 11is a diagram1100illustrating a configuration of image capturing system containing a mobile phone and a spherical camera in accordance with one embodiment of the present invention. Diagram1100illustrates a mobile phone116having a spherical camera166with multiple surface-mount lenses placed on the surface of camera166. Diagram1102illustrates a side view of mobile phone116shown in diagram1100. In one embodiment, spherical camera166is attached to mobile phone116via an expandable stick. Alternatively, a phone clip is used to assist the movement of camera166. Note that camera166can move up and down with respect to mobile phone116.

The camera can be a companion used within current mobile phones as it uses surface mount lenses and their special configuration which allows it for a small dimension that can be integrated as part of the phone itself. Since camera166is fitted with wireless communication hardware and can be controlled remotely, it can be mounted within proximity of the user, or to be far from the user to take picture(s). The 3DC system further allows camera166to be attached to other types of remote configurations such as a quadruple flying machines or drone to take pictures.

Having briefly described an aspect of 3DC system using a portable processing device and a camera,FIG. 12illustrates an exemplary portable device or mobile phone having a digital processing system capable of processing 3D/360 images in accordance with one embodiment of the present invention. It will be apparent to those of ordinary skill in the art that other alternative network or system architectures may also be employed.

Computer system1200, which can be applied to a mobile phone, includes a processing unit1201, an interface bus1212, and an input/output (“TO”) unit1220. Processing unit1201includes a processor1202, main memory1204, system bus1211, static memory device1206, bus control unit1205, and mass storage memory1207. Bus1211is used to transmit information between various components and processor1202for data processing. Processor1202may be any of a wide variety of general-purpose processors, embedded processors, or microprocessors.

Main memory1204, which may include multiple levels of cache memories, stores frequently used data and instructions. Main memory1204may be RAM (random access memory), MRAM (magnetic RAM), or flash memory. Static memory1206may be a ROM (read-only memory), which is coupled to bus1211, for storing static information and/or instructions. Bus control unit1205is coupled to buses1211-1212and controls which component, such as main memory1204or processor1202, can use the bus. Mass storage memory1207may be a magnetic disk, solid-state drive (“SSD”), optical disk, hard disk drive, floppy disk, CD-ROM, and/or flash memories for storing large amounts of data.

I/O unit1220, in one example, includes a display1221, keyboard1222, cursor control device1223, web browser1224, and communication device1225. Display device1221may be a liquid crystal device, flat panel monitor, cathode ray tube (“CRT”), touch-screen display, or other suitable display device. Display1221projects or displays graphical images or windows. Keyboard1222can be a conventional alphanumeric input device for communicating information between computer system1200and computer operator(s). Another type of user input device is cursor control device1223, such as a mouse, touch mouse, trackball, or other type of cursor for communicating information between system1200and user(s).

Communication device1225is coupled to bus1211for accessing information from remote computers or servers through wide-area network. Communication device1225may include a modem, a router, or a network interface device, or other similar devices that facilitate communication between computer1200and the network. In one aspect, communication device1225is configured to perform wireless functions.

3D/360 imaging processor1230, in one aspect, is coupled to bus1211within processing unit1201. In one example, 3D/360 imaging processor1230is configured to processing 3D/360 image information and generates 3D/360 images and videos based on the sensed information. 3D/360 imaging processor1230can be operated in hardware, software, firmware, or a combination of hardware, software, and firmware.

The exemplary aspect of the present invention includes various processing steps, which will be described below. The steps of the embodiment may be embodied in machine or computer executable instructions. The instructions can be used to cause a general purpose or special purpose system, which is programmed with the instructions, to perform the steps of the exemplary aspect of the present invention. Alternatively, the steps of the exemplary embodiment of the present invention may be performed by specific hardware components that contain hard-wired logic for performing the steps, or by any combination of programmed computer components and custom hardware components.

FIG. 13is a block diagram1300illustrating a process of handling captured images in accordance with one embodiment of the present invention. Diagram1300illustrates multiple cameras1302using multiple surface-mount lenses to capture images as well as audio or voice. Upon synchronizing multiple cameras at block1306, 3D/360 images generated in fisheye format is adjusted to form equirectangular format at block1308. Upon finding the 3D/360 views via feature finder at block1310, features are matched by a matcher at block1312. Upon inter-image alignment at block1314, the images or pixels are compensated by a lightening compensation process at block1316. After applying histogram equalization to the captured images at block1318, the pixels or images are at least partially stitched at block1320. Once the images or pixels are blended, seamed, or removed at block1322, the image(s) is forwarded to display1324for projecting 3D/360 image on a screen. If cameras1302record audio information, the audio information is forward to gain equalization component to process audio information at block1326. After removing noise from the audio information at block1328, the audio or voice data is forwarded to display1324to play audible sound based on the audio data.

FIG. 14is a flowchart1400illustrating a process of processing 3D/360 images using a spherical camera and a portable device in accordance with one embodiment of the present invention. At block1402, a process capable of capturing 3D/360 viewing image(s) receives a first lens orientation signal from a remote portable device such as a mobile phone via a communication network. At block1404, the first lens of multiple lenses mounted on a spherical surface of a camera is orientated in a first direction in accordance with the first lens orientation signal. After receiving a second lens orientation signal from the remote portable device via the communication network at block1406, the second lens of the multiple lenses mounted on the spherical surface of the camera, at block1408, is orientated in a second direction in accordance with the second lens orientation signal. At block1410, at least one 3D/360 image is captured by the first lens.