Patent ID: 12244964

DETAILED DESCRIPTION

I. Smartphone with Side-mounted Cameras

Reference is made toFIGS.1and2, which depict a smartphone100according to an exemplary embodiment. The smartphone100in this exemplary embodiment can be based upon an existing commercially available model from (e.g.) Apple, Samsung, LG, etc., or an updated design from any acceptable manufacturer. In general, the smartphone includes a housing110that is either sealed against the environment or porous, as appropriate. The housing110includes a rear face120(FIG.1) and opposing front face220(FIG.2). Each face can be (e.g.) 2-3 inches wide and 4-6 inches tall in a conventional implementation. The housing110has a peripheral side edge along its top (130), bottom (132), left (134) and right (136) sides, which defines a thickness TH of between approximately ⅛ inch and ½ inch in various examples. However, this measurement is highly variable, and can be more or less that the stated range. More generally, the housing face size and/or thickness can vary widely for differing makes/models of smartphone contemplated herein.

The side edges130,132,134,136can include various interface components—for example function buttons140,230,232,234, charging/data ports, audio in/out and speakers (not shown—along bottom edge132). These components can vary widely across different smartphone platforms and can be omitted in certain examples—where wireless data/charging is used and the housing is meant to be sealed against the environment. At least one, and possibly several, microphones250(FIG.2) are provided for audio input on the front face220. A main interface (home) button252can also be provided along the front face220in various examples.

Notably, the depicted smartphone100includes a plurality of cameras, located in a manner that produces coverage of an approximately 360-degree circle160about the longitudinal (long) axis HA of the housing110(SeeFIG.1). As such a rear camera170is located on the rear face120. This camera170can also serve as a primary high-resolution photography camera. A front camera260, which can also serve as a so-called selfie camera, is located on the opposing front face220(FIG.2). Moreover, side cameras180and182are located one each on opposing left and right side edges134and136, respectively. These side cameras180,182can be located at any point along the housing side edge. In this example, they are placed near to (e.g. within ½ and 1½ inches of the top edge130to avoid interference with the grasping fingers of a user. Other placements are expressly contemplated. The side cameras180,182can be constructed using conventional technology—for example small-scale board cameras of sufficient resolution. The lens for each camera can be sealed into a port on the side edge, or a separate sealed transparent cover can overlie an embedded camera lens. The side cameras can have fixed focus, mechanical autofocus (using e.g. a variable mechanical lens or liquid lens), or can rely on electronic focus. Each side camera can include an appropriate two-dimensional (2D) image sensor of appropriate pixel resolution or another mechanism for translating a scanned image of a 1D pixel array into a 2D image. The construction of the camera assembly is described further below.

The housing provides a front display screen222located along all or a portion of the front housing face220, as is known in smartphone technology (FIG.2). The housing also provides an optional rear display screen190located along all or a portion of the rear housing face120(FIG.1). The rear screen190can be embedded behind a transparent window or provided flush with the face120. Likewise, the screen can include conventional touchscreen functionality along all or a portion of its surface area, or can be a display-only screen. The screen190typically employs a color LCD matrix to provide images. The screen190and underlying rear face120can be constructed so as to be removable by (e.g.) a technician for servicing or replacement of components. The removability can be accomplished in a variety of ways that should be clear to those of skill. For example, a sealed seam around the screen perimeter can allow the rear face120to be separated from the rest of the housing110. The rear screen190can be connected to the main circuit board and power supply of the smartphone100using a conventional connection arrangement—for example a flexible ribbon cable—in a manner clear to those of skill.

With reference toFIG.3, the generalized construction of the smartphone100is shown in further detail.FIG.3is a cutaway top view of the multi-camera, multi-screen smartphone ofFIG.1, showing placement of the primary circuit board, cameras and screen with respect to the smartphone housing. The housing110is shown in phantom for further clarity. The main circuit board310is shown residing in the approximate center of the housing thickness TH, although actual placement in the housing is highly variable. Additionally, the main circuit board310can be substituted for one or more sub-assembly circuit boards as appropriate. As shown, the front touchscreen LCD matrix320is placed in a conventional location overlying the main circuit board, and is connected to it in a conventional manner. The front camera260, rear camera170, and side cameras180and182can all be connected directly or indirectly to the main circuit board310. The front and rear cameras260and170are shown connected directly to the main circuit board310. The side camera units180,182are shown positioned with respect to each side edge134,136of the housing110. In this example, the units180and182are provided on small rigid or flexible circuit boards340and342(respectively). The boards340and342are part of, or connected to, appropriate ribbon cables350and352, respectively. These cables350and352are connected to the main circuit board310and/or a sub-assembly circuit board. While not shown, a rechargeable battery is provided within the housing and interconnected to the circuit board and any appropriate power-handling circuitry. The rear display screen LCD matrix330is also shown connected to and/or overlying the main circuit board310.

The cameras170,180,182and260are all interfaced with one or more appropriate video processor(s), which can be instantiated on separate chip(s) or as part of the overall smartphone processor. This/these video processor(s) can convert the image data acquired by each camera into a data stream so that this data can be further handled for storage, transmission (e.g. streaming) and/or other processes in a manner clear to those of skill. One or more of these handling processes can be controlled by the video imaging and/or video chat application(s)192,194(FIG.1). The function of this/these applications is now described in further detail.

With reference toFIGS.1,2and4, the cameras located along the sides120,134,136and220of the housing110include optics that are arranged to provide coverage over a substantial portion, or entirety of, a 360-degree circle160. As shown, the cameras each define a field of view (in the horizontal plane perpendicular to the phone's longitudinal (long) axis HA), which is represented by associated viewing angles θA, θB, θCand θD. In the example ofFIG.4, each camera of the smartphone100includes a portion of a 360-degree scene. The field of view (θA-θD) of each camera can be fixed, or can vary (using known electronic and physical zoom/focus-adjustment) to focus upon the subject(s) A, B C and D in each field. The image data from each camera is input to the camera processor and handled by the video imaging and chat applications192,194(FIG.1). These applications allow the streams to be packetized and transmitted over (e.g.) a wireless LAN and/or cellphone network (WAN)420to one or more recipient smartphone(s) or other computing device(s)430. The recipient phone430includes a version of the chat application, and can display the subjects A, B, C and D in an associated set of windows432,434,436and438. An optional rear display generally in accordance with this description can also be provided on the recipient phone430. Alternatively, the audio input(s)440,442of the smartphone100can triangulate upon the source of any sound from subjects A, B, C and D, and only display on the recipient phone430the video stream for that/those subject(s) who are making sufficient noise. The decision to project the image data stream for a given subject can be based upon a decision made by the chat application in the smartphone100using conventional techniques that determine the relative direction from which a sound emanates. In one example, this operation entails using a stereo microphone arrangement or multiple microphones (e.g. inputs440and442) placed around the smartphone housing110. Each microphone receives a greater or lesser volume of a similar sound input, allowing the direction of the sound to be determined. More generally, the recipient can arrange his/her display to decide which subjects to display on his her phone, and how they will be arranged on the window.

The user of the smartphone100can also arrange displays of information or video from the recipient or his/her own cameras as shown. For example,FIGS.1and2each show a split screen with different subjects196and198(FIG.1) and270and272(FIG.2). The arrangement of subjects and split screens on the front and/or rear display is arbitrary and can be controlled automatically by the chat application, or based on settings entered by the user into the chat application interface. A common setup by the user would entail sending images of people around the recipient's room to split screens on the rear display and (optionally) the front display so that all parties on the user side could view all parties on the recipient side. Likewise, the recipient would project images of all parties on the user side on the rear, and optionally, front screens of the recipient's smartphone. This arrangement creates a fully immersive video chat experience for all parties on both sides.

Note that split screens can be substituted with another form of display imagery—for example, a blended image with various subjects blended together into a single screen. In another implementation the cameras can be used by the user to generate a fully panoramic (e.g. 360-degree) movie. On-board video editing functions can be used to blend the edges of each video stream to create a continuous effect. The 360-degree movie can be viewed using a variety of display modalities that should be clear to those of skill. One possible display function entails linking the imaging and/or video chat applications to the smartphone's built-in accelerometers and allow the image to switch as the smartphone is turned so the that rear screen now faces the user and the front screen faces to audience.

Note that an appropriate case or cover can be constructed using skill in the art to accommodate the additional cameras and rear display screen. Such case or cover can be watertight—using transparent windows where appropriate, or porous, using open ports to allow access for cameras and displays.

FIGS.5A-5Dshow graphical user interfaces on the front and back of a user's phone and a recipient's phone showing a video chat session, according to an illustrative embodiment.FIG.5Ais an exemplary graphical user interface on the front screen of user's multi-camera, multi-screen smartphone. A user's front screen222can show images collected from one or more of the cameras on a recipient's phone, and can show a small picture-in-picture image taken from the user's front camera. By way of non-limiting example, image tile502can show video recorded by the recipient's front camera, image tile504can show video recorded by the recipient's rear camera, image tile506can show video recorded by the recipient's left side camera, and image tile508can show video recorded by the recipient's right side camera. However, it should be clear that the arrangement and number of tiles can be varied. For example, if only one, two, or three cameras of the recipient's phone are imaging participants, one, two, or three corresponding image tiles can be displayed on the user's front screen222. An image tile512can be presented as a picture-in-picture (PIP) showing the video being collected from the user's front camera and sent to the recipient's phone. A user can optionally rearrange the placement of tiles, if desired, for example, by clicking and dragging any of the tiles502,504,506,508, or512into a different location on the front screen222. A user can optionally turn on or off cameras on the user's phone, or tiles on the user's screen to highlight selected participants.

FIG.5Bis a graphical user interface on the rear screen of the user's multi-camera, multi-screen smartphone showing a chat session with a recipient multi-camera, multi-screen smartphone, according to an illustrative embodiment. A user's rear screen190can show images collected from one or more of the cameras on a recipient's phone, and can show a small picture-in-picture image taken from the user's rear camera. By way of non-limiting example, image tile502can show video recorded by the recipient's front camera, image tile504can show video recorded by the recipient's rear camera, image tile506can show video recorded by the recipient's left side camera, and image tile508can show video recorded by the recipient's right side camera. However, it should be clear that the arrangement and number of tiles can be varied. For example, if only one, two, or three cameras of the recipient's phone are imaging participants, one, two, or three corresponding image tiles can be displayed on the user's rear screen190. An image tile514can be presented as a PIP showing the video being collected from the user's rear camera and sent to the recipient's phone. A user can optionally rearrange the placement of tiles, if desired, for example, by clicking and dragging any of the tiles502,504,506,508, or514into a different location on the rear screen190.

FIG.5Cis a graphical user interface on the front screen of a recipient's multi-camera, multi-screen smartphone showing a chat session with the user's multi-camera, multi-screen smartphone, according to an illustrative embodiment. A recipient's front screen520can show images collected from one or more of the cameras on a user's phone, and can show a small picture-in-picture image taken from the recipient's front camera. By way of non-limiting example, image tile512can show video recorded by the user's front camera, image tile514can show video recorded by the user's rear camera, image tile516can show video recorded by the user's left side camera, and image tile518can show video recorded by the user's right side camera. However, it should be clear that the arrangement and number of tiles can be varied. For example, if only one, two, or three cameras of the user's phone are imaging participants, one, two, or three corresponding image tiles can be displayed on the recipient's front screen520. An image tile502can be presented as a PIP showing the video being collected from the recipient's front camera and sent to the user's phone. A recipient can optionally rearrange the placement of tiles, if desired, for example, by clicking and dragging any of the tiles512,514,516,518, or502into a different location on the front screen520. A recipient can optionally turn on or off cameras on the recipient's phone, or tiles on the recipient's screen to highlight selected participants.

FIG.5Dis a graphical user interface on the rear screen of recipient's multi-camera, multi-screen smartphone showing a chat session with the user's multi-camera, multi-screen smartphone, according to an illustrative embodiment. A recipient's rear screen522can show images collected from one or more of the cameras on a user's phone, and can show a small picture-in-picture image taken from the recipient's rear camera. By way of non-limiting example, image tile512can show video recorded by the user's front camera, image tile514can show video recorded by the user's rear camera, image tile516can show video recorded by the user's left side camera, and image tile518can show video recorded by the user's right side camera. However, it should be clear that the arrangement and number of tiles can be varied. For example, if only one, two, or three cameras of the user's phone are imaging participants, one, two, or three corresponding image tiles can be displayed on the recipient's rear screen522. An image tile504can be presented as a PIP showing the video being collected from the recipient's rear camera and sent to the user's phone. A recipient can optionally rearrange the placement of tiles, if desired, for example, by clicking and dragging any of the tiles512,514,516,518, or504into a different location on the rear screen522.

FIG.5Eis a graphical user interface on the front screen of a user's multi-camera, multi-screen smartphone including multiple PIPs showing images from the user's cameras, according to an illustrative embodiment. In various embodiments, in addition to showing one or more image tiles showing video recorded from one or more cameras on the recipient's phone, the front screen222and/or rear screen of a user's phone can show one or more PIPs showing images recorded by the user's front, rear, left side, and/or right side cameras. By way of non-limiting example, front screen222can include the recipient's front camera image tile502with a PIP of the user's front camera image tile512, the recipient's rear camera image tile504with a PIP of the user's rear camera image tile514, the recipient's left camera image tile506with a PIP of the user's left camera image tile516, and the recipient's right camera image tile508with a PIP of the user's right camera image tile518. However, it should be clear that in various embodiments, fewer than all cameras or tiles may be used, and that the arrangement of tiles can be varied. Similarly, multiple tiles and PIPs can be presented on the user's rear screen, the recipient's front screen, and/or the recipient's rear screen.

FIG.5Fis a graphical user interface on the front screen of a user's multi-camera, multi-screen smartphone including multiple mini-tiles showing images from the user's cameras, according to an illustrative embodiment. In various embodiments, in addition to showing one or more image tiles showing video recorded from one or more cameras on the recipient's phone, the front screen222and/or rear screen of a user's phone can show one or more mini-tiles showing images recorded by the user's front, rear, left side, and/or right side cameras. As shown inFIG.5F, mini-tiles of the image tile512showing the user's front camera, image tile514showing the user's rear camera, image tile516showing the user's left side camera, and image tile518showing the user's right side camera can be presented together in a mini-tile stripe524at the bottom of the screen, so that each tile is presented separately. Tiles can be presented so that no tile overlaps with, or interferes with another tile. The mini-tiles showing the images collected from the user's cameras can be shown without (free of) blocking other image tiles. In various embodiments, the mini-tile can be arranged in a mini-tile stripe524at the top of the screen as shown inFIG.5G, or in a mini-tile stripe in the middle of the screen as shown inFIG.5H, without (free of) blocking other image tiles.

In various embodiments, the mini-tiles can be arranged in a vertical mini-tile strip524along the left side of the screen, as shown inFIG.5I, in a vertical mini-tile strip524down the center of the screen, as shown inFIG.5J, or in a vertical mini-tile stripe524along the right side of the screen, as shown inFIG.5K. It should be clear that any of the above arrangements of tiles can also be implemented on the user's rear screen, the recipient's front screen, and/or the recipient's rear screen.

FIG.5Lis a graphical user interface on the front screen of a user's multi-camera, multi-screen smartphone with a mini-tile stripe and a single video image from the recipient's camera, according to an illustrative embodiment. A user with a multi-camera, multi-screen smartphone can receive a single video stream from a single camera, such as the front camera, of the recipient's phone. A user's screen can have an image tile502showing video recorded by the recipient's front camera and a mini-tile stripe524, or an image tile504showing video recorded by the recipient's rear camera along with a mini-tile stripe524, or various other combinations of video streams shown on the user's screen522. It should be clear that the user's screen(s) can have one or more image tiles from cameras on the recipient's phone, along with one or more images from cameras on the user's phone that can optionally be arranged in a mini-tile stripe.

FIG.6briefly describes a procedure600for operating a multi-camera, multi-display smartphone and video chat application according to an exemplary embodiment. As shown, the user operates his or her smartphone imaging and/or video chat application so that the smartphone's operating system begins any communication and related operations. This can include notifying the recipient smartphone (over the wireless network) that the transmitting user smartphone wishes to have a chat (step610). The user smartphone can automatically activate the cameras that surround the housing—or these can be activated manually. The activation can include a motion detection (or IR-sensing) function that only operates cameras on which active subjects are present (step620). Then, the camera(s) acquire images of the subjects within their field of view and pick up any accompanying audio. One, two, three, or four camera(s) can acquire images of subjects, and various cameras can acquire images of different subjects that are in different positions around the longitudinal axis of the phone. This data is translated into one or more image data streams, with appropriate (e.g.) mpeg data compression, etc., as should be clear to those in the art. The video/audio data streams are packetized and transmitted over the network to the recipient smartphone (step630), where they are received by the recipient and displayed in a manner desired by the recipient's chat application (e.g. multiple, split screens, a blended image, etc.). The recipient's screen can display the images from the one, two, three, or four cameras showing the imaged subjects together on a single screen, using for example, a split screen or tiled screen. Likewise, the user smartphone receives one or more image data streams from the recipient in step640, and these streams are displayed to the user and/or his/her audience in an appropriate manner using the front and rear displays in step640. The user's smartphone can show the same display on both the front and the back screen, so that each of the subjects can see the image data stream(s) being sent from the recipient phone. The nature of the display on the user's smartphone can be controlled via manual settings and/or automatic processes (block650).

FIG.7is a perspective view of a smartphone with cameras located on the front, back, and both sides of the smartphone, according to an illustrative embodiment. A user's smartphone700can show two or more video image data streams taken from two or more cameras on the recipient's smartphone. As shown inFIG.7, the upper image data stream702can be taken from the recipient's rear camera, and the lower image data stream704can be taken from the recipient's front camera, however, the arrangement of images data streams can be varied and can be adjustable by the user. Multiple image streams, such as upper image data stream702and lower image data stream704can be shown on the front screen701and/or the rear screen709. The same image data streams can be shown on both screens.

The user's smartphone700can have a front camera706and a rear camera708. The user's smartphone can also have cameras in one or more positions along the side of the phone.FIG.8is a front view of the smartphone with cameras located on the front, back, and both sides of the smartphone, according to an illustrative embodiment. Turning toFIGS.7and8, the smartphone700can have side cameras710and712located in the upper portion of the smart phone, and in various embodiments the smartphone can also have side cameras730and732located in the lower portion of the smartphone, and/or side cameras720and722located in additional locations along the side of the smartphone, such as the central area of the smartphone.FIG.7depicts three cameras along each side, with a camera in the upper portion, a camera in the central portion, and a camera in the lower portion, however, it should be clear that in various embodiments, different numbers of cameras and/or different arrangements of cameras along the side of the smartphone are also possible. Having multiple cameras along the sides of the phone can allow the chat app to select different cameras to obtain the best view of the chat participants. Different handhold positions, or different types of tripods or other phone holders, can all block different cameras while leaving other cameras exposed to collect video image data of the surrounding scene. Having multiple cameras on each side allows the chat app to collect video data from exposed cameras while other cameras may be blocked.

FIG.9is a rear view of the smartphone with cameras located on the front, back, and both sides of the smartphone, according to an illustrative embodiment.FIG.10is a left side view, andFIG.11is a right side view of the smartphone with cameras located on the front, back, top, bottom, and both sides of the smartphone, according to an illustrative embodiment. Side cameras710,712,720,722,730, and732can be located in various positions along the sides of the smartphone, and the smartphone can have three, more than three, or fewer than three cameras along each side.

FIG.12is a perspective view of a smartphone with cameras located on the front, back, top, bottom, and both sides of the smartphone, according to an illustrative embodiment. A user's smartphone1200can show two or more video image data streams taken from two or more cameras on the recipient's smartphone. As shown inFIG.12, the upper image data stream1202can be taken from the recipient's rear camera, and the lower image data stream1204can be taken from the recipient's front camera, however, the arrangement of images data streams can be varied and can be adjustable by the user. Upper image data stream1202and lower image data stream1204can both be displayed on front screen1201.

The user's smartphone1200can have a front camera1206and a rear camera1208. The user's smartphone can have side cameras1210in one or more positions along the side of the phone. The user's smartphone can have one or more top cameras1240along the top of the smartphone, and one or more bottom cameras along the bottom of the phone.FIG.13is a front view of the smartphone with cameras located on the front, back, top, bottom, and both sides of the smartphone, according to an illustrative embodiment. Turning toFIGS.12and13, the smartphone1200can have two or more side cameras1210and1212along the sides of the smartphone. The smartphone can have one or more top camera1240and one or more bottom camera1242. In various embodiments, different numbers of cameras and/or different arrangements of cameras along the top and bottom of the smartphone are also possible. Having cameras1240and1242along the top and bottom of the smartphone can allow the chat app to collect video image data from all directions around the smartphone, including both sides, when the smartphone is being held sideways. That is to say, when the smartphone is held in a lengthwise or landscape orientation, the top and bottom cameras can collect video image data from areas at the sides of the smartphone, so that the chat app can see all of the 360 degrees surrounding the smartphone. The top camera1240and bottom camera1242can also be referred to as side cameras. Top camera1240can be referred to as a right side camera, and bottom camera1242can be referred to as a left side camera.

FIG.14is a rear view of the smartphone with cameras located on the front, back, top, bottom, and both sides of the smartphone, according to an illustrative embodiment. The smartphone1900can have a rear screen1209on the back of the smartphone.FIG.15is a top view, andFIG.16is a bottom view of the smartphone with cameras located on the front, back, top, bottom, and both sides of the smartphone, according to an illustrative embodiment.FIG.17is a right side view, andFIG.18is a left side view of a smartphone with cameras located on the front, back, top, bottom, and both sides of the smartphone, according to an illustrative embodiment. Two or more side cameras1210and1212, one or more top cameras1240, and one or more bottom cameras1242can be located along the periphery of the smartphone. The smartphone can have more cameras than shown, and the arrangements of cameras can be different than shown. In various embodiments, cameras can be located in various positions around the peripheral edge of the smartphone, including the top, bottom, sides, corner regions between the top and the sides, corner regions between the bottom and the sides, or various positions around the edges of the smartphone. Cameras in different positions around the smartphone can allow the chat app to see all sides of the smartphone regardless of the orientation of the smartphone.

FIG.19Ais a front view of a smartphone with multiple cameras, according to an illustrative embodiment. Smartphone1900can have a front screen1902, front camera(s)1906, side cameras1910,1912,1920,1922,1930,1932, top camera1940, and bottom camera1942. Cameras can be located on the front, rear, sides, top, and bottom, along with various corner regions, etc. Cameras can be flush with the edge of the phone, such as cameras1920and1922, or can be inset such as cameras1910and1912. Cameras can be inset by an inset distance ID that can be in a range between 0 and 5 mm. Cameras can point directly out, or can be cocked or angled. Some cameras, such as cameras1930and1932can be broad-angle lenses, or fish-eye lenses to capture broad ranges of a scene. Different types of cameras, cameras in different positions, cameras at different angles or insets, etc., can allow the chat app to have access to a variety of different video image data streams that can each be useful in different situations.

FIG.19Bis a front view of a smartphone with multiple wide-angle cameras, according to an illustrative embodiment. Smartphone1950can have a front screen1951, front camera(s)1956, side cameras1960and1962, top camera1980and bottom camera1987. Cameras can be located on the front, rear, sides, top, and bottom, along with various corner regions, etc. In various embodiments, cameras such as cameras1960and1962can be inset into notches1964and1966. Cameras1960and1962can have lenses1970and1972that can extend into notches1964and1966. In various embodiments, lenses1970and1972can be domed lenses, or bubble lenses, or fish eye lenses. Because lenses1970and1972extend outward into the notches1964and1966, they can enable cameras1960and1962to capture images greater than 180 degrees around the sides of the phone.

In various embodiments, cameras such as top camera1980can have a lens1982extending into notch1984. Notch1984can be covered and/or filled-in with a clear shell1986. The clear shell1986can allow the top and/or sides of the phone to have a smooth, continuous appearance, while also defining a notch that the lens can extend into to allow the lens to focus light from the imaged scene in an arc that is greater than 180 degrees around the sides of the phone. In various embodiments, cameras such as bottom camera1987can have a flush covering lens1988in the notch. The covering lens1988can flushly fill the notch so that the side edge of the phone case defines a smooth, continuous appearance. Appropriate covers and cases can be designed to facilitate this form factor.

With more particular reference toFIG.19Can enlarged perspective view of the bottom camera ofFIG.19Bis shown. The smartphone1950includes the above-described camera lens1987within the notch1989. The smartphone1950can also have a covering lens1988that allows the camera to have a field of vision greater than an arc of 180 degrees. This lens1988can be constructed to include various optical effects—for example, internally reflective, fish eye-lensmatic and/or prismatic effects—to allow the light to bend to, or beyond an arc of 180 degrees. The lens1988can be clear or tinted and conform closely to the edge profile of the surrounding phone case. In fact, in various embodiments, the lens1988can be tinted so as to emulate the color of the case (using (e.g.) appropriate dye and/or flash coating techniques). In this manner, the lens can blend almost seamlessly into the surrounding case edge, whist defining optics that provide the 180-degree-plus viewing arc. Note that the lens and cover arrangement1987and1988ofFIGS.19B and19Ccan be placed at any location (or plurality of locations) around the perimeter of the camera case as appropriate to provide the desired imaging effect—for example, it is expressly contemplated that a version of the flush-fitting arrangement can be located on each opposing side edge as described above. Thus the depiction of the positioning of the lens/cover arrangement along the bottom edge of the case should be taken as exemplary. As noted above, an outer protective shell or case can include appropriate windows and/or slots to allow passage of light into the camera in manner clear to those of skill.

FIG.19Dis front view of the side of a smartphone with an extending camera, according to an illustrative embodiment. In various embodiments, a smartphone1950can have a top camera1981that can be within a slight bulge1983that extends out from the top of the phone. The bulge1983can have a curved outer surface. In a non-limiting example, the bulge can extend out from the top of the phone by a camera height CH of up to approximately 5 mm. The camera1981that is in a bulge1983extending up from the phone can collect image data from more than 180 degrees around the camera. It should be clear that any of the cameras and/or lenses described herein, including recessed cameras, flush cameras, extending cameras, bubble lenses, prismatic lenses, etc. can be used in any location on the phone, and can be used in different combinations with other cameras and lenses.

FIG.19Eis a schematic, top-view diagram showing the acquisition of multiple still and/or streaming images using cameras with overlapping fields of view, according to an illustrative embodiment. Side cameras1960and1962can have fields of view1990and1992with viewing angles θEand θG. Viewing angles θEand θGcan be up to or more than 180 degrees, so that the viewing angles can overlap. Front camera1956and rear camera1954can have fields of view1996and1994; with viewing angles θFand θH. Fields of view1994and1996that can overlap with fields of view1990and1992, so that the cameras can produce a complete 360 coverage around the smartphone1950. The chat app can integrate images from various cameras in different locations around the smartphone, such as cameras1954,1956,1960, and1962to create a seamless picture and/or video with 360 degree coverage. The chat app can have the location and field of view of each camera calibrated relative to the location and field of view of neighboring cameras, so that the images from each camera can be stitched together with its neighboring cameras to create a seamless 360 degree picture/video. By way of non-limiting example, images from side camera1960can be stitched together with images from front camera1956and rear camera1954to create a seamless image. Images from other cameras can be stitched together with neighboring cameras in a similar manner to create a complete 360 degree coverage.

II. Conferencing and Associated Multi-Pane Displays

FIG.20is a schematic view of four people seated around a smartphone and participating in a video chat with a remote recipient, and showing participants displayed on the remote participant's smartphone, according to an illustrative embodiment. Participant A, Participant B, Participant C, and Participant D can all participate together in a video chat with a remote participant. A smartphone1900can be held by hand, or held in a tripod, stand, or other device so that the smartphone is positioned between various video chat participants. Cameras in various positions around the phone, including a front camera1906, rear camera1908, and side cameras1910and1912, can collect video image data of the various participants and stream the video image data to the remote recipient's smartphone screen2002. Remote recipient's smartphone screen2002can display an upper image data stream2010showing participants A, B, C, and D, and a lower image data stream2012showing the remote participant on his own screen. The remote participant is labeled participant E. As shown inFIG.20, participant A is speaking, and so participant A is shown as larger than other participants and/or centered in the video being displayed in the upper image data stream2010. In various embodiments, the chat app can identify which participant is speaking and can increase the image size and/or center the image of the participant who is speaking. The chat app can identify the speaking participant by identifying the direction of the sound, and/or by analyzing the video image to identify which participant is speaking.

FIG.21is a schematic view of four people seated around a smartphone and participating in a video chat with a remote recipient, and showing participants displayed on the remote participant's smartphone in a tiled arrangement, according to an illustrative embodiment. The remote participant's smartphone screen can display the participants participating at the user's smartphone in a compact tiled display2102, with four rectangular tiles arranged in a rectangular grid. Various arrangements are possible, and can be varied depending on the number of participants, the preferences of the user, etc. The remote participant may also add his own video image data stream to his screen as a picture within a picture, or as another tile located somewhere on his own screen. The remote participant's tile can be the same size or a different size than the tiles for the participant's from the user's phone, and can occupy a separate part of the screen or can overlay other tiles. Similarly, the user's smartphone screen can show the remote participant, and can also show the local participants A, B, C, and D who are located around the user's smartphone in various arrangements. The remote and local participants can be arranged on the screen in any arrangement as described within this application.

FIG.22is a schematic view of four people seated around a smartphone and participating in a video chat with a remote recipient, and showing participants displayed on the remote participant's smartphone in another tiled arrangement, according to an illustrative embodiment. The remote participant's smartphone screen can display the participants participating at the user's smartphone in a long tiled display2202, with each participant arranged in a row. The participants can be arranged in the same order that they are arranged around the table. The remote participant can hold the remove phone sideways to allow the remote participants to be shown side-by-side at a maximum possible size. Similarly, the local user can hold the phone sideways to allow imaging of multiple remote participants to be shown in a row. While holding the phone sideways, cameras at the top and bottom of the phone can be referred to as side cameras, and can collect video image data of participants located to the sides of the phone. Video image data of participant(s) at a particular phone can also be shown on the screen for that phone.

FIG.23is a schematic view of four people seated around a smartphone and participating in a video chat with a remote recipient, and showing participants displayed on the remote participant's smartphone in a panoramic arrangement, according to an illustrative embodiment. The remote participant's smartphone screen can display the participants participating at the user's smartphone in a panoramic format, with the participants arranged as they are arranged around the user's smartphone. The chat app can combine, or stitch together, video image data from different cameras to show all participants at the user's smartphone next to each other in a panoramic view as they appear around the table. Four people seated around a table, as shown, can appear unfolded on the panoramic display2302as four people sitting in a row. The chat app can include a module that calibrates the location of each of the cameras relative to each other, so that the chat app can combine the images correctly into a single panoramic view. In various embodiments, the chat app can rely on the manufacturer's specifications for the locations of each camera, and/or can use the areas of overlap between cameras to calibrate the cameras relative to each other.

III. Conferencing Stand

FIG.24is a schematic view of a stand for a smartphone, according to an illustrative embodiment. A smartphone stand2400can be designed to hold a phone up in the middle of a table, in the center of a conference, in the center of a group of participants, etc. The smartphone stand2400can be designed to minimize the amount of the phone that is covered by the grasping mechanisms of the stand. In various embodiments, the phone can be inserted into the stand up to stand depth SD, which can be less than two inches, less than 1.5 inches, or less than 1 inch. Stand2400can have a cradle2402, a first support2404, and a second support2406. The first and second supports2404and2406can have pads2408at the end of the support, so that the pad can rest against the phone. Pads2408can be made of a resilient material. The second support2406can be biased towards the smartphone1200by biasing member2412, which can be a spring. The second support2406can have a release mechanism2410, including a fingergrip2414, so that a user can squeeze on the fingergrip2414to compress the biasing member2412and rotate the second support2406around the pivoting hinge2430. The stand2400can have a stem2420and a base2424. Base2424can be a suction cup, so the stand can be stuck to a table or other surface. In various embodiments, the base can also be a threaded member adapted to attach to a tripod, or a weighed base, or other bases that can be used to support a smartphone. In various embodiments, a stand2400can have multiple bases that can be interchangeable.

IV. Case with Side-Mounted Cameras

Reference is made toFIGS.25-28, which show a smartphone (or other mobile device, such as a tablet) arrangement2500having an external, removable (or permanent) accessory case2510attached to the housing of a device (smartphone)2520. The case2510can be formed from any acceptable material, such as PCC, EVA and/or silicone and includes appropriate shoulders, rims, etc., to provide an interference or friction-fit attachment to the device2520. Alternatively, snaps, fasteners, adhesives, etc. can be used to secure the case2510to the device2520in a manner generally clear to those of skill. The exemplary device2520includes at least one (e.g.) front camera2810(shown in phantom inFIG.28), and (optionally) at least one rear camera2710(shown in phantom inFIG.27). Note that various current, commercially available devices can include multiple cameras on a given side. The case2510includes appropriate (optional) windows or ports2712to allow the rear camera(s)2710and any associated illumination, microphones, etc. to be exposed to the environment. As described below, the case2510can include an alternate, or supplemental rear face camera.

The case2510is constructed with integral wells or pockets that receive appropriate embedded circuitry2610(shown as generalized circuit components) that facilitate operation of one or more camera(s)2530and associated optics (lenses, covers, seals, etc.) on each of opposing side edges2532of the case2510. The electronics can include any appropriate conventional or custom components that allow a miniaturized image sensor to acquire images (as described above) within a respective FOV relative to the side of the case. Optional illumination assemblies, and/or other conventional components, can be included in the electronics and project through the side. A sealed window2534can cover the camera components and (optionally) the illuminator to provide a weatherproof and continuous outer surface. The case can also include an (optional) microphone disposed in an associated hole/recess2536relative to each of the cameras2520, so as to provide directional pickup of sound within the associated camera field of view (FOV).

The case2510includes an opening and/or port2730at an appropriate location along its (e.g.) bottom, to allow access to a charging port and/or data interconnect (for example a an Apple Lightning® connector or micro USB on the base of the device (smartphone, tablet, etc.)2520. This interconnects with an integrated or separate connection lead2740attached to the circuitry2610of the case. This lead2740can deliver power between the case and the phone. For example, the case2510can be powered by the device batteries, or can have its own battery pack mounted in an electronics pod2750along the case back and/or another location in the case. The case and phone can also share power with batteries in each unit. The lead2740facilitates transfer of image data from the case camera(s)2530to the processing arrangement of the device2510, and provides control signals from the device to the case circuitry2610via an appropriate interface and control application2820, which can be stored on the device and operated by its processor. The application operates in a manner described generally above and allows the two (or more) case-based cameras2530to operate in conjunction with one or both (or more) of the device-based rear and front cameras2710and2810. This allows at least four different, concurrent views which can be used with conferencing, photography, and video applications, as described above.

In a particular example, the case electronics define a system having at least one proprietary USB hub (USB 3.0 standard, for example)2650(FIG.26) with service ports for one to three externally mounted UVC camera devices, which can include the side-mounted cameras2530. The hub2650is connected via the lead2740to the USB-C OTG/Charging port on Android style cell phones via a standard USB-C type connector. Alternatively, the hub2650can be adapted to operate with a Lightning® based port in conjunction appropriate interface software and firmware, which can be implemented in accordance with skill in the art. In an exemplary embodiment, the hub2650and cameras2530receive their (e.g.) 5 volt power supply from the cell phone battery and do not require external power.

The interface and control application2820is adapted to detect and connect to onboard (2530) and external (2710,2810) camera devices to capture and display all video feeds concurrently. The video feeds from each of the cameras can be characterized as discrete, virtual feeds, which are then amalgamated by the application and associated software/hardware into a single outbound video stream, coupled with audio and then streamed via (e.g.) RTMP/WebRTC and/or other standard data transfer protocols. Point-to-point or broadcast connections enable users to connect directly to other users, or broadcast one-to-many via an HTTP based streaming service. A multi-pane display with the various feeds(s) can be provided to the device screen and/or any receiving user's screen according to any of the arrangements described above.

FIG.29shows an exemplary arrangement2900that employs the general principles of a multi-camera case as described herein. The device2920(e.g. an Android-style smartphone) is secured within a case2910in accordance with the embodiments herein. The case2910includes integrally mounted cameras2930and2932and associated control/driving circuits, including a hub. In this example the cameras2930and2932are mounted on each of opposing sides of the case2910. Note that top and/or bottom side cameras (not shown) can also be included on the case in various embodiments, with an appropriate hub and control arrangement. The front device camera2922is also shown. The case-based side cameras2930and2932each acquire images within a respective, opposing FOV, and these images are converted to a stream that is projected, for example, on the device screen as panes2940and2942. The screen can also project the device camera feeds as desired.

The case in the exemplary embodiments, and other embodiments contemplated herein, can define any acceptable shape or form factor. It can be adapted to directly fit a particular device, or the case can define a two-piece structure having a single (outer) module that contains electronics, cameras, etc., in conjunction with a lower cost (inner) adapter that connects the module to a device of a given shape. In this manner, only the adapter need be provided in various shapes and the functional electronics module can be largely universal to various types of devices. Likewise, the adapter can be designed to allow attachment of electronics “pods” that carry out the functionality herein. Note that the case of the embodiments herein can be designed to including protective padding and/or weatherproofing components in various embodiments so as to also act as a protective case for the device in addition to a part of an overall imaging and communication system.

Note also that camera optics for each of the side camera can vary in a manner described above—for example, the optics can facilitate wide-angle views, mechanical and/or electronic zoom and other special effects (e.g. near infrared or other non-visible light sensing). Illuminators (if provided) can be adapted to project in similar non-visible wavelengths and/or visible wavelengths.

V. With reference to the exemplary arrangement3000ofFIG.30a device case3010, which is constructed and operated according to any of the arrangements described above, can be adapted to allow external cameras and other devices with appropriate data interfaces to be linked through a smartphone (or other device)3020via one or more integrated (or external) hub(s)3030and at least one (e.g. USB connector). Note that the case in such embodiments can be substituted with, or supplemented with, another interconnected component, such as a separate interface pod with an associated hub, that is, in turn, connected to the smartphone/device3020. More generally, the multi-port hub arrangement3030, can reside partly or fully within the internals of phone housing itself, or can be integrated with the removable or semi-removable (i.e., detachment requires multiple steps) case, or can be part of another type of housing, such as a detached, detachable, or removable/semi-removable pod. In an embodiment, the hub3030can be used without (free of) additional (e.g., side) cameras to enhance the overall usefulness and versatility of the device. Such a hub pod can be removably/semi-removably secured to the phone housing using clips, hook-and-loop fasteners, adhesives, etc. It employs an appropriate application as described herein to coordinate data feeds from the various connected peripheral device(s). Commercially available (non-proprietary), or proprietary, connectors can reside between the hub pod and the device. The hub pod provides the capability in any of the embodiments herein of attaching one or more non-camera, compatible peripheral devices, as described further below.

Hence, the case3010or similar component can be employed to interface with a variety of peripheral devices that are relevant to a particular task—for example, in the case of the depicted medical scenario, the interconnected peripheral devices can include one or more endoscope(s)3040, medical telemetry devices3042, including, but not limited to, EKG, EEG, temperature monitor, lab assay, blood pressure monitor, venous/pulse oximetry, etc., environmental sensors and controls3044and/or auxiliary camera(s)3046. The device(s) can also receive control and/or feedback data from the smartphone (or other portable data communication device) using the bi-directional capability of the hub architecture. Control signals and data are transmitted over respective wired (or wireless with appropriate interfaces) links, where they are combined as a data feed (with video and other telemetry data in an appropriate format) by the hub3030. The combined feed is transmitted to the device3020based upon the interface application3060operating on the device, and optionally, within the circuitry of the case3010. The application3060can pass data (double arrow3062) from the feed to a data handling application3064. The data handling application3064can be adapted to parse various forms of data so as to derive useful information and deliver it to the user as a display on the local device (as shown) or to a remote user via the communication network (arrow3070). By way of non-limiting example, the application can be developed in (e.g.) Android Studio using the Java language. In operation, the application is adapted to listen on all available USB channels and initiate bidirectional, non-proprietary communication with peripheral devices. This arrangement allows the application to be vender-neutral and/or agnostic to the type and format of data. The data handling application can be tailored to the particular peripheral devices being interconnected and/or can contain a library of peripheral types and models so as to properly interpret data. Likewise, the system can allow for download and loading in the device3010of specific functional modules that allow processing of a peripheral' device's data from the stream.

With reference toFIG.31, the user's device screen3110is divided into (e.g.) four viewing panes3140,3142,3144and3146that can be of similar or unequal size—for example, an image can be larger than an alphanumeric readout. These panes can correspond to the respective outputs of peripheral devices3040,3042,3044and3046. The depicted form of each output is only exemplary and this form, i.e. image, alphanumerics, graph and video, can vary widely, and be modified by the user if options for a different presentation are provided in the data handling application3064(or other application).

It should be clear that the type and form of displayed device outputs in various panes can vary widely and panes can be provided for non-peripheral-based data, such as patient chart information, browsers, data entry, etc. This arrangement3000makes possible an effective remote data center that a first responder or remote practitioner can use to consult with a base of operation over any distance globally.

In alternate examples, peripheral devices can be relevant to differing tasks, such as thermal imagers, night vision scopes, volt-ohm meters, robotic cameras, and the like, in (e.g.) a construction or recovery site. Various law enforcement and military applications are also facilitated, including connectivity with vital sensors, communication links and body/helmet cameras. Likewise, in warehouse environments and/or logistics applications, the device can be employed to interconnect barcode scanners and remote cameras. So long as a particular peripheral is adapted to transmit data in a form receivable by the hub, then it can be added to a feed for subsequent division and processing into an appropriate viewing pane on the local and/or remote device.

VI. Conclusion

It should be clear that the multi-camera, multi-screen smartphone (or other device) as described above, provides a versatile and user-friendly platform for acquiring video images of room full of subjects for use in conducting an immersive chat or a creating panoramic movie. The arrangement of cameras effectively employs the available real estate of the front, rear and sides of the phone, in combination with available miniaturized components, to provide the desired functionality. A variety of software applications can be developed to take full advantage of the power of such a smartphone system. Additionally, the functional arrangement of cameras is also advantageously implemented using a case that is removably attached to the smartphone, thus allowing for a conventional front and rear-camera smartphone to have the functionality described herein. The electronics and hub arrangement can be extended to use with other peripheral devices that provide enhanced awareness for a variety of fields and tasks—such as medicine.

The side cameras, in conjunction with the front and rear cameras, along with screens on both sides of the device can allow families to sit in comfortable and informal settings in various locations around a room and still participate in a single video chat together as a group. Each participant can be within the field of view of at least one camera, although not all participants need to be in the field of view of the same camera, and each participant can see one of the screens, although not all participants need to see the same screen. In this way, families, groups of friends, business meeting participants, or others can participate in a video chat together without (free of) crowding around a single camera and without (free of) crowding around a single screen.

The foregoing has been a detailed description of illustrative embodiments of the invention. Various modifications and additions can be made without departing from the spirit and scope of this invention. Features of each of the various embodiments described above may be combined with features of other described embodiments as appropriate in order to provide a multiplicity of feature combinations in associated new embodiments. Furthermore, while the foregoing describes a number of separate embodiments of the apparatus and method of the present invention, what has been described herein is merely illustrative of the application of the principles of the present invention. Also, as used herein, the terms “process” and/or “processor” should be taken broadly to include a variety of electronic hardware and/or software based functions and components (and can alternatively be termed functional “modules” or “elements”). Moreover, a depicted process or processor can be combined with other processes and/or processors or divided into various sub-processes or processors. Such sub-processes and/or sub-processors can be variously combined according to embodiments herein. Likewise, it is expressly contemplated that any function, process and/or processor herein can be implemented using electronic hardware, software consisting of a non-transitory computer-readable medium of program instructions, or a combination of hardware and software. Additionally, as used herein various directional and dispositional terms such as “vertical”, “horizontal”, “up”, “down”, “bottom”, “top”, “side”, “front”, “rear”, “left”, “right”, and the like, are used only as relative conventions and not as absolute directions/dispositions with respect to a fixed coordinate space, such as the acting direction of gravity. Additionally, where the term “substantially” or “approximately” is employed with respect to a given measurement, value or characteristic, it refers to a quantity that is within a normal operating range to achieve desired results, but that includes some variability due to inherent inaccuracy and error within the allowed tolerances of the system (e.g. 1-5 percent). For example, it is expressly contemplated that the software application (instantiated on the phone and/or a remote server) controlling the operation of the multiple cameras, and/or display of images therefrom, can allow for a plurality of smartphones to be used in combination—for example at spaced apart locations along a tabletop, with camera images from both sets of cameras appropriately displayed to a remote viewer. Such an application can, thus, include appropriate handshake protocols to link the communications of two cameras in a conference-like communication arrangement. Moreover, while this description describes handheld “smartphones” of various sizes, the same multiple-camera functionality can be implemented on slightly larger handheld devices—such as so-called tablets—for example the popular iPad from Apple—and the description should be taken broadly to include such devices and the common ranges of dimensions in which such devices are manufactured. Accordingly, this description is meant to be taken only by way of example, and not to otherwise limit the scope of this invention.