Source: https://patents.google.com/patent/US10268761B2/en
Timestamp: 2019-09-20 14:50:57
Document Index: 208333464

Matched Legal Cases: ['Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 61', 'Application No. 2012105639027', 'Application No. 2012105639027']

US10268761B2 - Panoptic visualization document collection - Google Patents
Panoptic visualization document collection Download PDF
US10268761B2
US10268761B2 US13/414,940 US201213414940A US10268761B2 US 10268761 B2 US10268761 B2 US 10268761B2 US 201213414940 A US201213414940 A US 201213414940A US 10268761 B2 US10268761 B2 US 10268761B2
US13/414,940
US20130166997A1 (en
2011-12-21 Priority to US201161578349P priority Critical
2011-12-21 Priority to US201161578361P priority
2011-12-21 Priority to US201161578357P priority
2012-03-08 Application filed by Boeing Co filed Critical Boeing Co
2012-03-08 Priority to US13/414,940 priority patent/US10268761B2/en
2012-03-08 Assigned to THE BOEING COMPANY reassignment THE BOEING COMPANY ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: Eames, Patrick J, HADLEY, BRENT L
2013-06-27 Publication of US20130166997A1 publication Critical patent/US20130166997A1/en
2019-04-23 Publication of US10268761B2 publication Critical patent/US10268761B2/en
A panoptic visualization document collection system includes a document parser and a colligater, and is configured to collect components of a document collection in a manner that that captures logical relationships between their content, which may in turn reflect relationships between the subjects or their objects depicted by the content. These logical relationships may be established by links between document components, which may be identified by information provided in associated metadata for the document components. The content may then be exploited according to these relationships, such as in a panoptic arrangement of the content in a manner that reflects the relationships. This type of arrangement may enable a user to see the content in a single view, and in an arrangement that facilitates a better understanding of the relationships between the content and their subjects or objects, which may be otherwise difficult to comprehend.
The present application claims priority to U.S. Provisional Patent Application No. 61/578,361, entitled: Panoptic Visualization Document Collection, filed on Dec. 21, 2011. The present application is related to U.S. Provisional Patent Application No. 61/578,349, entitled: Panoptic Visualization Document Layout, and U.S. Provisional Patent Application No. 61/578,357, entitled: Panoptic Visualization Document Navigation, both of which filed on Dec. 21, 2011. The present application is also related to U.S. patent application Ser. No. 13/072,217, entitled: Image Management and Presentation, filed on Mar. 25, 2011, which claims priority to U.S. Provisional Patent Application No. 61/436,585, entitled: Navigation to Aircraft Maintenance Information Using Tiled 2D Illustrations, filed on Jan. 26, 2011. The contents of all of the aforementioned are incorporated herein by reference in their entireties.
Example embodiments of the present invention are generally directed to a panoptic visualization document collection system, and corresponding method and computer-readable storage medium. In accordance with example embodiments, media content of a document collection may be collected in a manner that captures logical relationships between the content, which may in turn reflect relationships between the subjects or their objects depicted by the content. The content may then be exploited according to these relationships, such as in a panoptic arrangement of the content in a manner that reflects the relationships. This type of arrangement may enable a user to see the content in a single view, and in an arrangement that facilitates a better understanding of the relationships between the content and their subjects or objects, which may be otherwise difficult to comprehend. In the context of a complex system such as an aircraft including a number of components, subsystems and parts, the arrangement may not only lessen the time required to review documents containing large amounts of information, but may also lessen the time required and ease the difficulty studying the information to understand the ways in which the complex system and its elements relate to each other. Various ones of the relationships may be customizable, which when reflected in a panoptic arrangement, may further facilitate the understanding of various users. Example embodiments may therefore ease the difficulty and time required to search and/or navigate even the most massive amounts of information contained in some document collections.
According to one aspect of example embodiments, the system includes a document parser configured to receive and disassemble one or more electronic documents into a plurality of constituent document components each of which includes respective media content. The documents in one example are disassembled according to a predetermined type of the documents.
The system of this aspect also includes a colligater coupled to the document parser and configured to colligate the document components. In this regard, the colligation of the document components includes, for each of one or more document components of the plurality, the colligater being configured to identify one or more links between the document component and one or more other document components of the plurality. The link(s) of one example are identified according to the documents, type of documents, and/or media content of the document component and other document component(s).
In one example, a link of the link(s) is identified for the document component and other document component(s) sharing a common document with the respective document component, or for the document and other document component(s) adjacent the respective document component in a predetermined sequence.
In one example, each document component of the plurality includes media content having a subject and including one or more objects reflecting or forming the respective subject. In this example, the link(s) may be identified according to the subject or object(s) of the document component, and the subject or object(s) of other document component(s) of the plurality. In one further example, a link of the link(s) is identified for the document component and other document component(s) sharing a common subject or common object(s) with the respective document component. In another further example, a link of the link(s) is identified for the document component and other document component(s) including an object that is the subject of the respective document component, or other document component(s) having a subject that is an object of the respective document component.
In addition to identifying link(s), the colligation of document components also includes the colligater being configured to provide, in metadata associated with the document component, information identifying the link(s) between the document component and other document component(s). The colligater of this aspect is further configured to communicate the document components and metadata as a panoptic visualization document collection.
In one example, the colligation of document components may further include the colligater being configured to identify a layout for visual representations of at least some of the document components. And the colligater may be configured to provide, in metadata associated with each document component in the layout, information specifying a size, location and/or depth of the visual representation of the respective document component in the layout.
In one example, the system may further include a tiler coupled to the colligater. The tiler may be configured to generate, for each of one or more document components of the plurality, visual representations of the document component at respective resolutions for zoom levels of the document component. The tiler may be configured to divide the visual representations across the zoom levels into progressively larger numbers of tiles each of which covers a spatial area of the visual representation at a respective zoom level. In this example, the colligater being configured to communicate the document components may include being configured to communicate the document components to the tiler. And the tiler may be further configured to communicate the tiles of the visual representations of the document components.
In other aspects of example embodiments, a method and computer-readable storage medium are provided for collecting documents for panoptic visualization. The features, functions and advantages discussed herein may be achieved independently in various example embodiments or may be combined in yet other example embodiments further details of which may be seen with reference to the following description and drawings.
FIGS. 5-16, which schematically illustrate examples of suitable layout models according to example embodiments; and
FIGS. 17-21 depict example layouts of pages that may be displayed in a GUI that has a predetermined viewable area, and which may be navigated by a user, according to an example embodiment of the present invention.
Some embodiments of the present invention will now be described more fully hereinafter with reference to the accompanying drawings, in which some, but not all embodiments of the invention are shown. Indeed, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these example embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. For example, unless otherwise indicated, reference something as being a first, second or the like should not be construed to imply a particular order. Also, something may be described as being above something else (unless otherwise indicated) may instead be below, and vice versa; and similarly, something described as being to the left of something else may instead be to the right, and vice versa. Like reference numerals refer to like elements throughout.
Example embodiments of the present invention relate generally to panoptic visualization of documents and, in particular, to panoptic visualization of documents or their document components in a manner that reflects logical relationships between the documents/components. Example embodiments will be primarily described in conjunction with aerospace applications. It should be understood, however, that example embodiments may be utilized in conjunction with a variety of other applications, both in the aerospace industry and outside of the aerospace industry.
Referring now to FIG. 1, a panoptic visualization system 100 is illustrated according to example embodiments of the present invention. The system may include any of a number of different subsystems (each an individual system) for performing one or more functions or operations with respect to one or more electronic documents. As shown, for example, the system may include a document collection system 102, document layout system 104 and/or document navigation system 106. Although being shown as part of the panoptic visualization system, one or more of the document collection system, layout system and/or navigation system may instead be separate from but in communication with the panoptic visualization system. It should also be understood that one or more of the subsystems may function or operate as a separate system without regard to others of the subsystems. And further, it should be understood that the panoptic visualization system may include one or more additional or alternative subsystems than those shown in FIG. 1.
As described herein, an electronic document (or simply document) may be any electronic media content capable of being visualized in an electronic and/or printed (or printable) form. The media content of a document may include one or more of textual, graphical or other visual content such as still images, video or the like. The document may be of a number of different types of documents in which the type of document may be defined by one or more characteristics of the document such as its format, media content or the like. Examples of suitable types of documents include computer-aided design documents (e.g., CAD, CATDrawing, CATPart, CATProduct, CATProcess, cgr, DWG, DXF, DWF, etc.), text-based documents (e.g., ANS, ASC, DOC, DOCX, HTML, PDF, RTF, TXT, WPD, etc.), presentation documents (e.g., PDP, PPT, etc.), graphics documents (e.g., BMP, GTE, JPEG, JP2, PNG, PSD, PSP, RAW, TIFF, etc.), video documents (e.g., AVI, MPEG, QuickTime, WMV, etc.) or the like. Other examples of suitable types of documents include single or collections of legal documents (e.g., court decisions, briefs, patents, etc.), books, articles, web pages, screenshots, service bulletins, engineering diagrams, warranties, technical drawings, wiring diagrams or the like. And still further examples of suitable types of documents include data sets such as engineering design data, wiring data, troubleshooting data, business data or the like.
Example embodiments of the present invention may therefore collect pages of document(s) according to logical relationships between the pages, which in one example in the context of a complex system such as an aircraft, may reflect structural relationships between its elements. The pages may be panoptically arranged in a manner that reflects the logical relationships, and in various examples, structural relationships. The arrangement may enable a user to see the pages in a single view, and in a manner that facilitates a better understanding of relationships that may be otherwise difficult to comprehend. In the context of a complex system for example, the arrangement may not only lessen the time required to review documents containing large amounts of information, but may also lessen the time required and ease the difficulty studying the information to understand the ways in which the complex system and its elements relate to each other.
Reference will now be made to FIGS. 2, 3 and 4, which illustrate more particular examples of a suitable document collection system, document layout system and document navigation system, respectively, according to example embodiments of the present invention.
FIG. 2 illustrates a document collection system 200 according to one example embodiment. As shown, the document collection system may include a document parser 202 configured to receive and disassemble one or more electronic documents into a plurality of constituent pages (document components) each of which includes respective media content. The documents in one example may be disassembled according to the type of the documents (e.g., computer-aided design documents, presentation documents, graphics documents, video documents, legal documents, books, articles, web pages, engineering diagrams, technical drawings, wiring diagrams, etc.). The document parser may therefore also be configured to identify the type of the documents, such as based on the format of the documents, business rules or by user input.
The colligater 204 may be configured to communicate the pages and metadata as a panoptic visualization document collection, and communicate the collection to any of a number of different destinations. In one example, the colligater may be configured to communicate the pages and metadata to respective storage 206, 208 for later retrieval. The storage may be resident with the document collection system 200, or may be separate from and in communication with the document collection system. The pages and metadata may be formatted and stored in any of a number of different manners, and hence, their storage may be of any of a number of different types. Examples of suitable types of storage include file storage, database storage, cloud storage or the like. In various examples, before pages are stored in respective storage 206, the pages may be compressed or otherwise processed for easier storage and retrieval. As shown, for example, the system may include a tiler 210 coupled to the colligater, and to which the colligater is configured to communicate the pages. The tiler may be configured to generate, for each of one or more pages, visual representations of the page at respective resolutions for zoom levels of the page. As described herein, the page (or rather its visual representation) at each zoom level may be generally referred to as a “sub-image,” and may at times be considered a separate version or copy of the page. As suggested above, however, a sub-image of a page need not necessarily include a still image as the page may include other types of media content.
In one example, the sub-image at each zoom level may be half the resolution of the zoom level immediately below it, and twice the resolution of the zoom level immediately above it (the resolutions increase or decrease by a factor of two). In this example, the number of tiles in a zoom level l may be given by tl=t0×4l, in which l=0, 1, 2, . . . L, and t0 represents the number of tiles in the top zoom level (level 0). In a more particular example including four zoom levels 0, 1, 2 and 3 (L=3), and in which the top zoom-level sub-image is composed of 1 tile (t0=1), the first intermediate zoom-level (level 1) sub-image may be composed of 4 tiles, the second intermediate zoom-level (level 2) sub-image may be composed of 16 tiles, and the base zoom-level (level 3) sub-image may be composed of 64 tiles.
As indicated above, the layout models may include any of a number of different types of layouts for panoptically arranging pages. In the layout models, logical relationship(s) established by link(s) between pages may be expressed by the arrangement of pages, in either or both of location or depth (z-order). In one example, logical relationships may be expressed by the proximity of pages to one another in their locations, and/or in their relative depths. Additionally, one or more of the layout models may define or imply a navigation path between documents related to one another, and/or a load shape for loading pages of a layout generated according to the respective layout models. Reference will now be made to FIGS. 5-16, which schematically illustrate examples of suitable layout models. As shown, these examples include brickwall, partial brickwall, hierarchy, shape, center out, top-down/triangulated, center-out triangulated, size dominance, overlap through detail, master-detail through depth, load shape, facet, mixed-media/static-and-dynamic or the like.
FIG. 5 illustrates a brickwall layout model 500 according to one example embodiment in which pages 502 may be arranged in one or more rows and one or more columns. As shown, the brickwall layout model may be characterized by a consistent end-to-end orientation with the pages being consistent in size and/or aspect ratio along one or more rows and/or columns. This type of layout model may be used for general search results. In one example, the layout model may provide a horizontal orientation related to chapter location, with a vertical orientation associated with a progressive increase in page length.
Although not separately shown, a partial brickwall layout model may be considered a superset of the brickwall layout model 500. The partial brickwall layout model may be characterized by clusters of pages that may similarly be consistent in size and/or aspect ratio at least within respective clusters. In this layout model, the clusters may be related by associated metadata of the respective pages, such as by their media content, author, time of creation, last revision or the like.
FIG. 6 illustrates a hierarchy layout model 600 according to one example embodiment in which pages may be arranged in a hierarchy in location and/or size. A hierarchical relationship between pages (hierarchically-related pages) may be indicated by their associated metadata in a number of different manners. For example, a hierarchical relationship may be indicated by a parent-child relationship between pages. In another example, a hierarchical relationship may be indicated by an object-subject or subject-object relationship in which an object of one page higher in a hierarchy may be subject(s) of other page(s) lower in the hierarchy, and in which at least some of the respective other page(s) may be at the same level in the hierarchy. In this example, the subject of the one page may be a master view, and the subject(s) of the other page(s) may be detailed view(s) of object(s) of the master view. In yet another example, a hierarchical relationship may be indicated by a relationship in which one page higher in a hierarchy includes reference(s) or link(s) (e.g., citation, hyperlink, etc.) to other document(s) lower in the hierarchy, in which at least some of the respective other document(s) may be at the same level in the hierarchy.
In one example of the hierarchy layout model 600, pages higher in a hierarchy may be located above those lower in the hierarchy (top-down), and/or may be larger in size than those lower in the hierarchy. The pages in this layout model may not be consistent in size or aspect ratio. Those pages higher in the hierarchy may dominate those lower in the hierarchy, and in which pages lower in the hierarchy may be constrained in the x-direction by the width of pages higher in the hierarchy.
More particularly, for example, the highest page 602 in the hierarchy may be located at the top, and may be sized according to the page's full resolution with its native aspect ratio. The next-highest page(s) 604 in the hierarchy may be located immediately below the highest page at a size smaller than the highest page, and with an aspect ratio that constrains the next-highest page(s) collectively to the width of the highest page. This pattern may repeat for the third-highest page(s) 606 in the hierarchy below respective ones of the next-highest pages, for the fourth-highest page(s) 608 in the hierarchy, and so forth. This layout model may in one example provide a single page with related pages cited in the respective page below it.
FIG. 7 illustrates a shape layout model 700 according to one example embodiment in which the model mimics a physical structure to which the pages relate, such as an aircraft or the geography of the United States. In one example, pages 702 may be arranged within boundaries of a visual representation 704 of the physical structure (two or three-dimensional representation), where the location(s) of page(s) may coincide with relationship(s) their media content to the structure. For example, pages related to the wing tip area of an aircraft (e.g., by subject or object) may be located in the respective area of the visual representation of the aircraft. In another example, pages related to court decisions from different states of the United States may be located in areas of the visual representation of the United States coinciding with the states in which the decisions were rendered.
In the shape layout model 700, in various instances, one or more subsets of the pages may be arranged in clusters 706 at respective areas of the visual representation of the physical structure. In these instances, the clusters may each be arranged according to another of the layout models, both in terms of location and size of the pages of the cluster.
FIG. 8 illustrates a center-out layout model 800 according to one example embodiment. This layout model may be characterized by a dominating, centrally-located primary page 802—such as a page identified by a search engine (e.g., search engine 302) searching for media content. Pages 804 related to the primary page (e.g., hierarchically-related) may be located around the primary page in a 360-degree arrangement. The pages in this layout model may not be consistent in size or aspect ratio. In one example, this layout model may indicate that the primary page has a relation to pre (upper), post (lower) and contemporaneous (beside) pages. An example of this type of layout may be a collection of documents or web pages that have been cross-hyperlinked together with the pages being part of a document tree that leads to the center, primary page.
FIG. 9 illustrates a top-down/triangulated layout model 900 according to one example embodiment. This layout model may be similar to a combination of the brickwall layout model 500 and hierarchy layout model 600, and may be characterized by a top-down arrangement of hierarchically-related pages 902. Similar to the brickwall layout model, the pages in this layout model may be consistent in size and/or aspect ratio at least within pages at the same level of the hierarchy, which may indicate the importance of the pages relative to one another. Similar to the hierarchy layout model, in one example, pages higher in the hierarchy may be above (in the y-direction) pages lower in the hierarchy; but in contrast to the hierarchy layout model, the widths of the lower pages may not be constrained by the higher pages in the x-direction. In one example, the top-down/triangulated layout model may provide a single page as a node in a graph tree of related pages.
FIG. 10 illustrates a center-out triangulated layout model 1000 according to one example embodiment. This layout model may be characterized by a dominating, centrally-located primary page 1002, similar to the center-out layout model 800. In the center-out triangulated layout model, pages 1004 related to the primary page (e.g., hierarchically-related) may be located above and/or below the primary page. Also similar to the center-out layout model, the pages of the center-out triangulated layout model may not be consistent in size or aspect ratio. The center-out triangulated layout model may indicate that the primary page has a relation to the pre (upper) and post (lower) pages. An example of this type of layout may be a collection of documents or web pages that have been hyperlinked together with the upper pages being part of the document tree that leads to the center, primary page, and the lower pages leading from the primary page.
FIG. 11 illustrates a size-dominance layout model 1100 according to one example embodiment. This layout model may be similar to the center-out layout model 800. In this regard, the size-dominance layout model may be characterized by a dominating, centrally-located primary page 1102, and pages 1104 related to the primary page (e.g., hierarchically-related) located around the primary page in a 360-degree arrangement. This layout model, however, may include more than one primary page that may be the same size or different sizes, with two such pages 1102 a, 1102 b being shown. The pages in this layout model may not be consistent in size or aspect ratio. The size of the pages in this layout model may convey weight or relevance of the pages relative to one another. An example of this type of layout may be a collection of wiring diagrams that have associated reference(s) or link(s) on all four axes of at least the primary diagrams. The referenced/linked wiring diagrams/schematics may themselves include relationships to additional diagrams.
FIG. 12 illustrates an overlap-through-detail layout model 1200 according to one example embodiment. This layout model may be characterized by preservation of a hierarchical relationship between pages in the depth (z-order) in which they are arranged. In one example, a page 1202 higher in the hierarchy may overlay another page 1204 lower in the hierarchy, which in turn may overlay another page 1206 even lower in the hierarchy. In this layout, pages sharing one or more common objects or other media content may overlap 1208 so that the shared object(s)/other media content may at least partially spatially align across the pages.
Navigating a layout (or more particularly its visual presentation) may include zooming into or out of the presentation. Zooming into the layout according to the overlap-through-detail layout model 1200 may include increasing the size and, in various instances of one example, resolution of its pages. In this layout model, however, zooming into the layout may also include obscuring an overlaying page so that one or more overlaid pages (or rather the overlaid portions thereof) underneath it may come into view. Conversely, zooming out of a layout may also include showing an obscured overlaying page so that the respective page comes into view, and one or more overlaid pages (or rather the overlaid portions thereof) go out of view. In one example, obscuring a page may include increasing a transparency of the page (or decreasing its opacity) during a zoom-in, and showing a page may include increasing an opacity of the page (or decreasing its transparency) during a zoom-out.
In one example, as shown in FIG. 12, in an overlap-through-detail layout model 1200 including pages 1202, 1204, 1206, page 1202 may be obscured as a user zooms into the layout. At some point during the zoom-in, then, the respective page may no longer be viewable so that the layout appears to only include pages 1204, 1206. The zoom-in may further continue, obscuring page 1204 until it is no longer viewable, at which point the layout appears to only include page 1206. In these instances, obscured pages may remain in the layout, or the layout engine 308 may be configured to remove them from the layout.
Continuing the example of FIG. 12 in which only page 1206 is viewable and pages 1202, 1204 are obscured, page 1204 may be shown overlaying page 1206 as a user zooms out of the layout. At some point during the zoom-out, the respective page may be viewable so that the layout appears to include both pages 1204, 1206. The zoom-out may further continue, showing page 1202 until it is also viewable, at which point the layout appears to include pages 1202, 1204, 1206. In these instances, overlaid pages (or the overlaid portions thereof) may remain in the layout, or the layout engine 308 may be configured to remove them from the layout.
FIG. 13 illustrates a master-detail through depth layout model 1300 according to one example embodiment. This layout model is similar to the overlap-through-detail layout model 1200 in that a hierarchical relationship between pages may be preserved in their depth (z-order). In this layout model, however, an overlaid page (e.g., lower in the hierarchy) may be sized and arranged to be coincident with the entirety of an overlaying page (e.g., higher in the hierarchy).
Similar to the overlap-through-detail layout model 1200, navigation in the master-detail through depth layout model 1300 may include zooming into or out of the layout, which may further include obscuring or showing an overlaying page relative to an overlaid page. In the example shown in FIG. 13, navigating a layout including pages 1302, 1304, 1306 may include obscuring overlaying pages as a user zooms into the layout. Conversely, navigating the layout may include showing overlaying pages as the user zooms out of the layout. In this regard, implementing a zoom in the master-detail through depth layout model of multiple pages may be similar to implementing a zoom for a page including multiple sub-images.
FIG. 14 illustrates a load-shape layout model 1400 according to one example embodiment. This layout model may be similar to some of the other layout models (e.g., center-out, center-out triangulated) including a dominating, centrally-located primary page 1402, and pages 1404 related to the primary page (e.g., hierarchically-related) located around, above and/or below the primary page. The pages in this layout model may not be consistent in size or aspect ratio. The load-shape layout model may be characterized by the timing of loading the layout (or more particularly its visual presentation). In this layout model, the non-primary pages may be loaded first, followed by the primary page.
An example of the load-shape layout model 1400 may be pages of media content related in time where pages of older media content may be loaded before the primary page of the most-current media content. The timing of media content in this context may relate to a time-aspect of the subject(s) or object(s) of the media content. For example, pages (e.g., pages 1404) for the preceding history of a court decision may be loaded before the court decision (e.g., page 1402) itself. In another example, the timing of media content may relate to the time of creation of the content, such as in the case of loading a first draft and one or more changes or revisions to a document (e.g., pages 1404) before the final or current version of the document (e.g., page 1402).
FIG. 15 illustrates a facet layout model 1500 according to one example embodiment. The facet layout model may be characterized by the arrangement of pages 1502 in a three-dimensional (3D) environment, such as in the context of facets of a 3D structure such as a truncated icosahedron 1504. For example, each page may be created from a set of x, y, z coordinates with a known consistent unit vector for orientation. Each page may then be placed in the layout in relation to a viewer's perspective. This layout model may be visually presented in three dimensions, or may be translated to a corresponding two-dimensional (2D) arrangement, such as a translated truncated icosahedron 1506.
FIG. 16 illustrates a mixed-media/static-and-dynamic layout model 1600 according to one example embodiment. This layout model may be a subset of one or more other layout models in which the pages of a layout may include pages of mixed media types. For example, this layout model may include one or more static (“S”) pages such as in the case of textual content or still images, and one or more dynamic (“D”) pages such as in the case of video. FIG. 16 illustrates for example the mixed-media/static-and-dynamic layout model arranging mixed-media content in an arrangement similar to the size-dominance layout model 1100. As shown, this arrangement may include dominating, centrally-located primary pages 1602 a, 1602 b, and pages 1604 related to the primary pages (e.g., hierarchically-related) located around the primary pages.
The document navigation system 400 may also include a request interface 404 coupled to the navigation engine 402 and configured to receive a request according to one or more of the selected navigation options. The request interface may be configured to communicate the request to the navigation engine. The navigation engine may in turn be further configured to effect an adjustment of the visual representation of the layout in response to the request and according to the request, and communicate the adjustment such as to the aforementioned GUI. In one example, the navigation engine may be configured to effect the adjustment of the visual representation without adjustment of the layout itself. In another example, the navigation engine may be configured to effect the adjustment of the visual representation including adjustment of the layout.
In addition to increase or decrease in size/resolution of pages of a layout, for one or more layouts, the zoom option may implicate one or more other navigation options to further affect one or more of the pages. For example, the overlap-through-detail layout model 1200 and master-detail through depth layout model 1300 may implicate obscure/show options in which overlaying pages may be obscured or shown relative to overlaid pages during zoom-in and zoom-out, such as in the manner explained above with reference to FIGS. 12 and 13. It should be understood, however, that the obscure/show options may be utilized to obscure or show one or more pages of a layout independent of the zoom option, and independent of pages overlaying one another.
To further illustrate example embodiments of the present invention, reference will now be made to FIGS. 17-21, which depict example layouts of pages that may be displayed in a GUI that has a predetermined viewable area, and which may be navigated by a user, according to an example embodiment of the present invention. FIGS. 17-19 illustrate an example of a layout according to the hierarchy layout model 600 displayed in a GUI that has a predetermined viewable area 1700. As shown, a layout according to the hierarchy layout model may include a plurality of pages 1702, each of which in one example may be a sub-image of the page at a respective resolution. Various ones of the pages may have different resolutions in the layout, with page 1704 having a higher resolution than pages 1706, which in turn have a higher resolution than pages 1708, which again in turn have a higher resolution than pages 1710.
The pages may be located and/or sized in the layout according to logical relationship(s) between the pages. In the example shown, the pages 1702 are engineering drawings of an aircraft, and may have object-subject relationships. More particularly, for example, object(s) of a page on a level of the hierarchy may be subject(s) of pages below it in the hierarchy, the subject(s) in one example being additional detail regarding the object(s).
More particularly, for example, page 1704 may depict an exterior view of the entire aircraft. Pages 1706 may depict a nose section, a landing gear assembly, a window, an engine assembly and a tail assembly. Pages 1708 may depict additional detail about the content of respective ones of pages 1706, and pages 1710 may depict additional detail about the content of respective ones of pages 1708. In the viewable area 1700, however, pages 1708, 1710 may be presented at a resolution that causes their media content to be only partially understood or not understood at all by a user. Of course, in other example embodiments, pages 1708 and/or pages 1710 may be presented at a sufficient resolution to interpret substantially all of their media content.
The GUI may present one or more selected navigation options for navigating the layout of pages 1702. In this illustrative example, pan and zoom navigation options may be presented in the form of controls 1712 to move and/or increase the size of the pages in the viewable area 1700 to focus on a portion 1714 of the layout. In other words, the user may activate the aforementioned controls to move and/or zoom the layout to fill a greater portion of the viewable area of the GUI with a portion of the layout. FIG. 18 illustrates one example of the result of navigating the layout in this manner.
As shown in FIG. 18, as the user activates controls 1712 to focus on the portion 1714 of the layout, the size of pages 1704-1710 may increase which, in one example, may include replacing sub-images of pages 1704-1708 with corresponding sub-images at higher resolutions. The resolution of the sub-images may allow the user to interpret substantially all of the presented media content. In this view, however, pages 1710 may not be presented with sufficient resolution to be interpreted by the user, because the size of the respective pages may not have been increased to a level implicating a next sub-image. That is, even after having increased the size of the image, the size may still most closely approximate the same sub-image so as to not cause its replacement with the next sub-image at a higher resolution. In this illustrative example, the user may again activate controls 1712 to move and/or resize the view to focus on an even smaller portion 1802 of the layout.
One example of the result of navigating the layout to focus on portion 1802 is shown in FIG. 19. In one example, the viewable area 1900 of the GUI may resize to more fully accommodate the focused portion of the layout. As the user activates the controls 1712 to focus on the portion 1802 of the layout, the size of pages 1704-1710 may increase, which in one example, may now further include replacing a sub-image of page 1710 with a corresponding sub-image at a higher resolution. The media content of page 1710 may now be sufficient to be interpreted by the user. In one example in which only a portion of page 1704 is within the viewable area of the GUI, and in which the respective page is divided into tiles, only those tiles covering the viewable portion of the respective page may be retrieved and displayed.
FIG. 20 illustrates an example of a layout according to the brickwall layout model 500 displayed in a GUI that has a predetermined viewable area 2000. The layout may include pages 2002-2010. In the example shown, the pages of the layout are wiring diagrams of a wiring system, which may be related by a spatial sequence. In this regard, page 2002 may depict wire(s) that connect to wire(s) depicted in page 2004. Page 2004 may in turn depict wire(s) that connect to wire(s) depicted in page 2006, and so forth for wire(s) of page 2006 to those of page 2008, and wire(s) of page 2008 to those of page 2010.
Pages of the layout entirely within the viewable area 2000 of the GUI may form or otherwise define an area or region of interest 2012. In one example, pages within the region of interest may be presented with an increased resolution relative to any pages outside the region of interest. As shown, the region of interest may include pages 2002-2008, but because only a portion of page 2010 is within the viewable area of the GUI, page 2010 may be considered outside the region of interest (although it may still be considered within the viewable area). In this instance, page 2010 may be presented at a lower resolution than the resolution(s) of pages 2002-2008.
FIG. 21 illustrates an example of a layout according to the overlap-through-detail layout model 1200 displayed in a GUI that has a predetermined viewable area 2100. As shown, the layout includes pages 2102, 2104, and in which page 2102 may include a reference to page 2104. As shown, this hierarchical relationship may be maintained by the layout in which page 2104 (higher in the hierarchy) may partially overlay page 2102, and in one example, page 2104 may overlay page 2102 at the location of its reference in the respective page. As also shown, page 2104 may be presented smaller in size than page 2102, and may therefore be presented with a lower resolution than page 2102.
According to example embodiments of the present invention, the panoptic visualization system 100 and its subsystems including the document collection system 102, document layout system 104 and document navigation system 106 may be implemented by various means. Similarly, the examples of a document collection system 200, document layout system 300 and document navigation system 400, including each of their respective elements, may be implemented by various means according to example embodiments. Means for implementing the systems, subsystems and their respective elements may include hardware, alone or under direction of one or more computer program code instructions, program instructions or executable computer-readable program code instructions from a computer-readable storage medium.
Generally, an apparatus of exemplary embodiments of the present invention may comprise, include or be embodied in one or more fixed or portable electronic devices. Examples of suitable electronic devices include a smartphone, tablet computer, laptop computer, desktop computer, workstation computer, server computer or the like. The apparatus may include one or more of each of a number of components such as, for example, a processor (e.g., processor unit) connected to a memory (e.g., storage device). The processor is generally any piece of hardware that is capable of processing information such as, for example, data, computer-readable program code, instructions or the like (generally “computer programs,” e.g., software, firmware, etc.), and/or other suitable electronic information. More particularly, for example, the processor may be configured to execute computer programs, which may be stored onboard the processor or otherwise stored in the memory (of the same or another apparatus). The processor may be a number of processors, a multi-processor core or some other type of processor, depending on the particular implementation. Further, the processor may be implemented using a number of heterogeneous processor systems in which a main processor is present with one or more secondary processors on a single chip. As another illustrative example, the processor may be a symmetric multi-processor system containing multiple processors of the same type. In yet another example, the processor may be embodied as or otherwise include one or more application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or the like. Thus, although the processor may be capable of executing a computer program to perform one or more functions, the processor of various examples may be capable of performing one or more functions without the aid of a computer program.
As explained above, example embodiments of the present invention permit the collection, layout and/or navigation of a large number of pages (or documents). Example embodiments may allow a user to simultaneously and quickly view and visually search a large number of pages while using less bandwidth than retrieving all of the pages. In instances in which a user may generally have an idea of the appearance of page(s) of interest, or have an idea of a logical relationship between the respective page(s) and other pages, a layout of pages may be generated and/or navigated to allow the user to locate and use page(s) of interest. The user may view and navigate pages as though they were laid out in the physical world without the physical space requirement involved with large collections of pages.
receiving and disassembling one or more electronic documents into a plurality of constituent electronic document components each of which includes respective media content, the documents being disassembled according to a predetermined type of the documents;
colligating the document components, including for each of one or more document components of the plurality:
identifying one or more links between the document component and one or more other document components of the plurality, the one or more links being identified according to one or more of the documents, type of documents, or media content of the document component and one or more other document components; and in metadata according to some metadata schema and associated with the document component,
providing structured information identifying the one or more links between the document component and one or more other document components that establish one or more logical relationships therebetween;
communicating the document components and metadata as a panoptic visualization document collection;
receiving and storing the document components and metadata in a page storage and a metadata storage, respectively;
identifying a subset of the document components from the page storage based on the metadata in the metadata storage;
generating a layout of including a panoptic arrangement of images of document components including the subset of the document components according to the metadata associated therewith, the panoptic arrangement being two-dimensional and the one or more logical relationships being expressed in the panoptic arrangement by a difference in size or depth of at least some of the images relative to others of the images, the images of document components including images of the respective media content thereof;
selecting one or more navigation options according to the metadata associated with the subset of the document components; and
communicating the one or more navigation options to a graphical user interface for presentation to a user along with the layout, the one or more navigation options thereby being selectable by the user to request an adjustment of the layout.
2. The method of claim 1, wherein a link of the one or more links is identified for the document component and one or more other document components sharing a common document with the respective document component, or one or more other document components adjacent the respective document component in a predetermined sequence.
3. The method of claim 1, wherein each document component of the plurality includes media content having a subject and including one or more objects reflecting or forming the respective subject, and
wherein the one or more links are identified according to the subject or one or more objects of the document component, and the subject or one or more objects of one or more other document components of the plurality.
4. The method of claim 3, wherein a link of the one or more links is identified for the document component and one or more other document components sharing a common subject or one or more common objects with the respective document component.
5. The method of claim 3, wherein a link of the one or more links is identified for the document component and one or more other document components including an object that is the subject of the respective document component, or one or more other document components having a subject that is an object of the respective document component.
6. The method of claim 1, wherein colligating the document components further includes:
identifying a layout for visual representations of at least some of the document components; and in metadata associated with each document component in the layout,
providing information specifying one or more of a size, location or depth of the visual representation of the respective document component in the layout.
7. The method of claim 1, wherein the method further comprises for each of one or more document components of the plurality:
generating visual representations of the document component at respective resolutions for zoom levels of the document component; and
dividing the visual representations across the zoom levels into progressively larger numbers of tiles each of which covers a spatial area of the visual representation at a respective zoom level,
wherein communicating the document components includes communicating the tiles of the visual representations of the document components.
8. An apparatus comprising a processor and a memory storing executable instructions that in response to execution by the processor cause the apparatus to implement at least:
a panoptic visualization document collection system comprising:
a document parser configured to receive and disassemble one or more electronic documents into a plurality of constituent document components each of which includes respective media content, the documents being disassembled according to a predetermined type of the documents;
a colligater coupled to the document parser and configured to colligate the document components, including for each of one or more document components of the plurality, the colligater being configured to:
identify one or more links between the document component and one or more other document components of the plurality, the one or more links being identified according to one or more of the documents, type of documents, or media content of the document component and one or more other document components; and in metadata according to some metadata schema and associated with the document component,
provide structured information identifying the one or more links between the document component and one or more other document components that establish one or more logical relationships therebetween,
wherein the colligater is further configured to communicate the document components and metadata as a panoptic visualization document collection; and
a page storage and a metadata storage configured to receive and store the document components and metadata, respectively;
a panoptic visualization document layout system configured to identify a subset of the document components from the page storage based on the metadata in the metadata storage, and generate a layout including a panoptic arrangement of images of document components including the subset of the document components according to the metadata associated therewith, the panoptic arrangement being two-dimensional and the one or more logical relationships being expressed in the panoptic arrangement by a difference in size or depth of at least some of the images relative to others of the images, the images of document components including images of the respective media content thereof; and
a panoptic visualization document navigation system configured to select one or more navigation options according to the metadata associated with the subset of the document components, and communicate the one or more navigation options to a graphical user interface for presentation to a user along with the layout, the one or more navigation options thereby being selectable by the user to request an adjustment of the layout.
9. The apparatus of claim 1, wherein a link of the one or more links is identified for the document component and one or more other document components sharing a common document with the respective document component, or one or more other document components adjacent the respective document component in a predetermined sequence.
10. The apparatus of claim 1, wherein each document component of the plurality includes media content having a subject and including one or more objects reflecting or forming the respective subject, and
11. The apparatus of claim 10, wherein a link of the one or more links is identified for the document component and one or more other document components sharing a common subject or one or more common objects with the respective document component.
12. The apparatus of claim 10, wherein a link of the one or more links is identified for the document component and one or more other document components including an object that is the subject of the respective document component, or one or more other document components having a subject that is an object of the respective document component.
13. The panoptic visualization document collection system of claim 10, wherein at least one of the one or more electronic documents relates to a complex system composed of a plurality of elements, at least some of the plurality of elements being composed of a plurality of other elements, and
wherein at least one document component has the complex system as its subject and elements of the plurality of elements or other elements as objects, and at least one other document component has an element of the plurality of elements as its subject, and elements of the plurality of other elements as objects.
14. The apparatus of claim 1, wherein the colligater being configured to colligate the document components further includes being configured to:
identify a layout for visual representations of at least some of the document components; and in metadata associated with each document component in the layout,
provide information specifying one or more of a size, location or depth of the visual representation of the respective document component in the layout.
15. The apparatus of claim 1, wherein the memory stores further executable instructions that in response to execution by the processor cause the apparatus to further implement:
a tiler coupled to the colligater and configured to generate, for each of one or more document components of the plurality, visual representations of the document component at respective resolutions for zoom levels of the document component, and divide the visual representations across the zoom levels into progressively larger numbers of tiles each of which covers a spatial area of the visual representation at a respective zoom level,
wherein the colligater being configured to communicate the document components includes being configured to communicate the document components to the tiler, the tiler being further configured to communicate the tiles of the visual representations of the document components.
16. A computer-readable storage medium that is non-transitory and has computer-readable program code portions stored therein that, in response to execution by a processor, cause an apparatus to at least:
receive and disassemble one or more electronic documents into a plurality of constituent document components each of which includes respective media content, the documents being disassembled according to a predetermined type of the documents;
colligate the document components, including for each of one or more document components of the plurality, the apparatus being caused to:
provide structured information identifying the one or more links between the document component and one or more other document components, that establish one or more logical relationships therebetween;
communicate the document components and metadata as a panoptic visualization document collection;
receive and cause storage of the document components and metadata in a page storage and a metadata storage, respectively;
identify a subset of the document components from the page storage based on the metadata in the metadata storage; and
generate a layout including a panoptic arrangement of images of document components including the subset of the document components according to the metadata associated therewith, the panoptic arrangement being two-dimensional and the one or more logical relationships being expressed in the panoptic arrangement by a difference in size or depth of at least some of the images relative to others of the images, the images of document components including images of the respective media content thereof;
select one or more navigation options according to the metadata associated with the subset of the document components; and
communicate the one or more navigation options to a graphical user interface for presentation to a user along with the layout, the one or more navigation options thereby being selectable by the user to request an adjustment of the layout.
17. The computer-readable storage medium of claim 16, wherein a link of the one or more links is identified for the document component and one or more other document components sharing a common document with the respective document component, or one or more other document components adjacent the respective document component in a predetermined sequence.
18. The computer-readable storage medium of claim 16, wherein each document component of the plurality includes media content having a subject and including one or more objects reflecting or forming the respective subject, and
19. The computer-readable storage medium of claim 18, wherein a link of the one or more links is identified for the document component and one or more other document components sharing a common subject or one or more common objects with the respective document component.
20. The computer-readable storage medium of claim 18, wherein a link of the one or more links is identified for the document component and one or more other document components including an object that is the subject of the respective document component, or one or more other document components having a subject that is an object of the respective document component.
21. The computer-readable storage medium of claim 16, wherein the apparatus being caused to colligate the document components further includes being caused to:
22. The computer-readable storage medium of claim 16, wherein the computer-readable storage medium has further computer-readable program code portions stored therein that, in response to execution by the processor, cause the apparatus to further:
generate, for each of one or more document components of the plurality, visual representations of the document component at respective resolutions for zoom levels of the document component, and divide the visual representations across the zoom levels into progressively larger numbers of tiles each of which covers a spatial area of the visual representation at a respective zoom level,
wherein the apparatus being caused to communicate the document components includes being caused to communicate the tiles of the visual representations of the document components.
US13/414,940 2011-12-21 2012-03-08 Panoptic visualization document collection Active 2034-06-26 US10268761B2 (en)
US201161578349P true 2011-12-21 2011-12-21
US201161578361P true 2011-12-21 2011-12-21
US201161578357P true 2011-12-21 2011-12-21
US13/414,940 US10268761B2 (en) 2011-12-21 2012-03-08 Panoptic visualization document collection
KR1020120131718A KR20130072123A (en) 2011-12-21 2012-11-20 Panoptic visualization document collection
EP12195735.1A EP2608077A1 (en) 2011-12-21 2012-12-05 Panoptic visualization document collection
JP2012276348A JP6059976B2 (en) 2011-12-21 2012-12-19 Panoptic visualization document collection
BR102012032628A BR102012032628A2 (en) 2011-12-21 2012-12-20 compilation panoptic view documents
CN201210563902.7A CN103177070B (en) 2011-12-21 2012-12-21 Panoptic visualization document collection
US20130166997A1 US20130166997A1 (en) 2013-06-27
US10268761B2 true US10268761B2 (en) 2019-04-23
ID=47683437
US13/414,940 Active 2034-06-26 US10268761B2 (en) 2011-12-21 2012-03-08 Panoptic visualization document collection
US (1) US10268761B2 (en)
EP (1) EP2608077A1 (en)
JP (1) JP6059976B2 (en)
KR (1) KR20130072123A (en)
CN (1) CN103177070B (en)
BR (1) BR102012032628A2 (en)
CN104123619B (en) * 2014-08-08 2017-05-24 上海利驰软件有限公司 Based on the principle method for dwg format quickly generate wiring diagrams and system management
US20080288535A1 (en) 2005-05-24 2008-11-20 International Business Machines Corporation Method, Apparatus and System for Linking Documents
US7805666B2 (en) * 2005-05-31 2010-09-28 Sap, Ag Multi-layered data model for determining image choice across a set of audience-specific documents
CN101894145A (en) 2010-06-28 2010-11-24 华为终端有限公司 Document display method and device
2012-03-08 US US13/414,940 patent/US10268761B2/en active Active
2012-11-20 KR KR1020120131718A patent/KR20130072123A/en active IP Right Grant
2012-12-05 EP EP12195735.1A patent/EP2608077A1/en active Pending
2012-12-19 JP JP2012276348A patent/JP6059976B2/en active Active
2012-12-20 BR BR102012032628A patent/BR102012032628A2/en active Search and Examination
2012-12-21 CN CN201210563902.7A patent/CN103177070B/en active IP Right Grant
CN1794229A (en) 2004-12-20 2006-06-28 微软公司 Management and use of data in a computer-generated document
"About Deep Zoom Composer," Microsoft, Copyright 2011 [retrieved on Nov. 23, 2011] Retrieved using Internet <URL: http://msdn.microsoft.com/en-us/library/dd409068(d=printer).aspx; 4 pages.
"Airframe & Powerplant Mechanics General Handbook," US Department of Transportation Federal Aviation Administration, Chapter 2, pp. 35-52; Mar. 31, 1999.
"Data differencing," 2012, Wikipedia, The Free Encyclopedia, [retrieved on Jul. 9, 2012] Retrieved using Internet <URL: http://en.wikipedia.org/w/index.php?title=Data_differencing&printable=yes, 3 pages.
"Deep Zoom," Mar. 2008, Wikipedia, The Free Encyclopedia, [retrieved on Apr. 18, 2012] Retrieved using Internet <URL: http://en.wikipedia.org/wiki/DeepZoom; 4 pages.
"Futuristic software from Minority Report is real," published Jul. 23, 2012, [retrieved on Aug. 30, 2012] Retrieved using Internet<URL: http://www.foxnews.com/tech/2012/07/23/futuristic-software-from-minority-report-is-real/; 3 pages.
"Hard Rock Memorabilia," Copyright 2009, Hard Rock Cafe International, Inc., [retrieved on Apr. 18, 2012] Retrieved using Internet <URL: http://memorabilia.hardrock.com, 2 pages.
Chinese Office Action dated Dec. 26, 2016 in Chinese Application No. 2012105639027 filed Dec. 21, 2012.
EP Search Report dated May 15, 2012 for EP Appl. No. 12152591.9, 13 pages.
EP Search Report dated May 15, 2012 for EP Appl. No. 12152591.9, 13 pgs.
Hunter, "Theodolite, Documentation/Help/FAQ," Apr. 24, 2012, Hunter Research and Technology LLC [retrieved on Dec. 14, 2012] Retrieved using Internet <URL: http://hunter.pairsite.com/mobile/theodolite/help/, 6 pages.
International Search Report dated Dec. 21, 2012 for PCT/US2012/054808, 4 pages.
International Search Report dated Jan. 18, 2013 for PCT/US2012/058937, 4 pages.
Second Chinese Office Action dated Aug. 2, 2017 for Application No. 2012105639027.
Written Opinion dated Dec. 21, 2012 for PCT/US2012/054808, 6 pages.
Written Opinion dated Jan. 18, 2013 for PCT/US2012/058937, 9 pages.
CN103177070B (en) 2018-01-12
JP6059976B2 (en) 2017-01-11
US20130166997A1 (en) 2013-06-27
CN103177070A (en) 2013-06-26
EP2608077A1 (en) 2013-06-26
JP2013137762A (en) 2013-07-11
KR20130072123A (en) 2013-07-01
BR102012032628A2 (en) 2015-10-20
US8056019B2 (en) 2011-11-08 System and method for providing a dynamic user interface including a plurality of logical layers
Phelps et al. 1996 Toward active, extensible, networked documents: multivalent architecture and applications
JP2012511208A (en) 2012-05-17 A preview of the search results for the proposed narrowing terms and vertical search
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:HADLEY, BRENT L;EAMES, PATRICK J;REEL/FRAME:027826/0048