Source: https://patents.google.com/patent/US6441802
Timestamp: 2018-03-23 16:56:15
Document Index: 141041101

Matched Legal Cases: ['art 251', 'art 252', 'art 251', 'art 251', 'art 253', 'art 201', 'art 201', 'art 201', 'art 203']

US6441802B1 - Interactive control system having plural displays, and a method thereof - Google Patents
Interactive control system having plural displays, and a method thereof Download PDF
US6441802B1
US6441802B1 US09619647 US61964700A US6441802B1 US 6441802 B1 US6441802 B1 US 6441802B1 US 09619647 US09619647 US 09619647 US 61964700 A US61964700 A US 61964700A US 6441802 B1 US6441802 B1 US 6441802B1
US09619647
G05B2219/23131—Select on large display part of pictogram to show on display of used workstation
G05B2219/23152—Large and several smaller displays for each workstation, each own cursor on large display
This is a continuation of U.S. patent application Ser. No. 09/374,263, filed Aug. 16, 1999, now U.S. Pat. No. 6,100,857, which is a continuation of U.S. patent application Ser. No. 08/969,313, filed Nov. 13, 1997, which issued as U.S. Pat. No. 5,969,697, which issued Oct. 19, 1999, which is a continuation of U.S. patent application Ser. No. 08/230,369, filed Apr. 20, 1994 which is now abandoned.
However, because information displayed on the large display and information shown in displays at hand are independently controlled, a conventional system required a complex operation to provide the necessary information in connection with each other. For instance, when a warning lamp blinks on the large display, the operators must retrieve an image displaying control data for the warning from hundreds of images by selecting menu repeatedly. Therefore, there has been a problem of a delayed response to an emergency such as an abnormal condition occurrence or an accident.
When realizing such a man-machine interface as the one above described, an important point to be considered is that the large display is shared by a plurality of operators. A monitoring and controlling system is operated by collaboration of plural operators, each of them is in charge of a different operation respectively, such as an operator in charge of operation, an operator in charge of maintenance and inspection, and a chief on duty for controlling total operation. Accordingly, the large display is shared by operators who perform different tasks simultaneously, which is different from a case of display at hand which is prepared for individual operators. Therefore, the above described interface must satisfy the following requirements.
(1) No Disturbance to Other Operator's Operation
(2) Simple Retrieval of Information Necessary for Individual Tasks by Respective Operators
(3) An Operating Environment Suitable for Tasks Assigned to Each Operator
Commands used frequently and permission for operation differ depending on the task changed to respective operator. Accordingly, it is desirable that the operating environment such as a structure of menu and an operable range of operation can be customized for respective operators.
In accordance with the present invention, the above described objects can be realized by providing a registering means for registering an attribute of a respective operator to an input-means, a process selecting means for selecting process contents based on the attribute responding to a process request from the input means, and an executing means for executing a process selected by the process selecting means and outputting to an output means selected based on the attribute, to an interactive processing apparatus having a plurality of input means and a plurality of output means.
Input information from the input apparatus 255 is transmitted to the process executing part 251 via the input/output processing part 252. The process executing part 251 executes a responding process and outputs results of the execution to the output apparatus 256 when the input information is such as to designate a position in a display of the output apparatus 256. On the other hand, when the input information is such as designating a position in a display of the output apparatus 205, the process executing part 251 transmits the input information with an attribute called out from the attribute memory part 253 to the process executing part 201 of the workstation 200. The process executing part 201 executes a process corresponding to the transmitted input information and the attribute. The process executing part 201 transmits results of the execution to an output apparatus corresponding to the transmitted attribute which is called out from the attribute memory part 203.
FIG. 2 illustrates an example of display manner on a large display 1. On the large display 1, overview information such as a system diagram 5 and warnings 4 on the plant is displayed. When an abnormal condition of the plant is detected, a warning 4 related to the abnormal condition blinks. Especially, when the abnormal condition is serious, warning sound alarms from the headsets 14, 34, 54 in addition to the blinking of the warnings 4. The numeral 3 designates a display controlling menu for controlling the display on the large display 1. By selecting the display controlling menu 3, the large display can display various images such as (1) prior displayed objects, (2) subsequent display objects, (3) weather information, (4) monitoring video image, (5) various system diagrams, etc. Each of pointers 15, 35, 55 works with each of the mice respectively, and/or is colored with different colors so as to facilitate identification. The pointers 15, 35, 55 naturally may have shapes different from one another or may be added with information on the operator's attribute, such as name of work, or personal name, instead of color coding. The pointers 15, 35, 55 can be transferred continuously between the displays at hand 10, 30, 50 and the large display 1, respectively. The transferring of the pointers will be explained in detail referring to FIGS. 4-6 later.
FIG. 3 illustrates an example of display manner on the display at hand 10. Hereinafter, an embodiment is explained taking the display at hand 10 as an example, but of the explanation can be applied to other displays at hand 30, 50 if any exception is not mentioned especially. FIG. 3 indicates an example of detailed information on the plant displayed on the display at hand 10. On the display at hand 10, a plate hanger icon 17 and a person in charge icon 16 are displayed. The plate hanger icon 17 provides a function to put a memorandum by voice to the display at hand 10 and the large display 1. The person in charge icon 16 provides a function to register charged task (operator, inspector, chief on duty, etc.) of an operator who uses input/output apparatus at hand, such as display at hand 10, a mouse 12, a keyboard 13, and a headset 14. On the person in charge icon 16, the registered charged task, that means the charged task to the operator who uses the display 10 at the time, is displayed. In the case shown in FIG. 3, “operator” is displayed on the icon 16. It means an operator in charge of operation is registered.
Next, a method for transferring the pointer 15 between the large display 1 and the display at hand 10 is explained referring to FIGS. 4-6. In the present embodiment, the pointer moves continuously from the display at hand to the large display 1 only by moving the mouse 12 forward as shown in FIG. 4. That means, under a condition wherein the pointer is located on the display at hand 10, the pointer 15 moves toward upper portion of the display at hand 10 in accordance with moving the mouse 12 forward, and reaches finally at the uppermost point of the display at hand 10. If moving the mouse 12 forward further, the pointer 15 transfers to the lowest point of the large display 1, and the pointer 15 moves upward to the top of the large display in accordance with further moving the mouse 12 forward. On the contrary, if the mouse 12 is moved backward under a condition wherein the pointer 15 is displayed on the large display 1, the pointer 15 moves downward to the lowest point of the large display 1. If moving the mouse 12 backward further, the pointer 15 transfers to the uppermost point of the display at hand 10. Display position of the pointer 15 at a moment when the pointer 15 transfers from the display at hand 10 to the large display 1 is decided as shown in FIG. 4. That is, when putting x for a horizontal position of the pointer 15 at the moment of transferring from the display at hand 10 to the large display 1, h for the number of pixels in a horizontal direction on the display at-hand 10, X for a horizontal position of the pointer 15 at the moment of entering into the large display 1, and H for the number of pixels in a horizontal direction, the X is decided so as to be x:h=X:H. Similarly, when the pointer transfers from the large display 1 to the display at hand 10, x is decided so as to be x:h=X:H.
Referring to FIG. 4, a method for transferring the pointer 15 as if the top portion of the display at hand 10 joins the total span of the lower portion of the large display 10 has been explained. However, a method for transferring the pointer 15 as if the top portion of the display at hand 10 joins the partial span of the lower portion of the large display can also be useful. A range of the partial lower span of the large display to be joined to the display at hand 10 can be decided in consideration of a relative relationship of an arrangement between the large display 1 and the display at hand 10. That means, when the display at hand 10 is arranged at left side to the large display 1, X and H are substituted by (X−d1) and (H−d1−d2), respectively, so that the pointer 15 transfers within a range d1 the at left side of the lower portion of the large display. Here, d2 is a distance from the right side of the large display 1 to right side of the display at hand 10 when the right side of the display at hand 10 is arranged at the right side of the large display 1. On the contrary, when the display at hand 10 is arranged at the right side of the large display 1, the pointer 15 is arranged so as to transfer at a right side range of the bottom of the large display 1.
The image size (the number of pixels) of the pointer 15 may be changed in the large display 1 from that in the display at hand 10, especially, when the display at hand 10 and the large display are installed far apart. Making the display size of the pointer 15 larger facilitates identification. For instance, the pointer 15 is displayed with pixels 16×16 on the display at hand 10, and with pixels 36×36 in the large display 1. According to the above selection, the pointer becomes easily recognizable even in the far away large display.
(1) a position on the display at hand 10 and the large display 1 can be designated continuously without changing grip of a pointing device,
(2) interactive operation of the large display can be performed with the same feeling as that of operation of the display at hand 10,
When a desired item in the charged task selecting menu 18 is selected by the mouse 12 (step 101), a password input region 19 is displayed. When a password which is designated to each of the charged tasks is input (step 102), the charged task for the operator is registered to the system, and the registered charged task is displayed in the charged task icon 16. When the charged task must be changed, the same procedure as that of the registering (FIG. 8) is performed.
FIG. 11, shows an example of displaying information 22 in the displays at hand 10, 30, 50 responding to pointing on the large display 1. However, the information can be output in sound. Even in a case when the information is output in sound, the information is output to only a person who needs the information. For instance, when the operator in charge of operation points out a display on the large display 1 by the mouse 12 at hand, the information related to the display is output to the headset 14 provided to the operator in charge of operation in sound. Furthermore, not only information, but also sound feed-back to the operation on the large display 1 is output to only the operator. For instance, when a sound signal is fed back at every pointing, the feed-back is output to only the headset 14 for the operator who has pointed, but not to the headsets for the other operators. That means, when the operator in charge of operation points a display on the large display 1 by the mouse 12 at hand, a sound signal is output to the headset 14 provided to the operator in charge of operation.
FIG. 12 illustrates an example of changing an arranging order of the menu items corresponding to the charged task of the operator. The numeral 22 indicates a menu displayed when the operator in charge of operation must point out any one of symbols in the system diagram 5 displayed on the large display 1 by the mouse 12. By selecting the menu 22, any one of the information related to the pointed symbol, such as data setting, monitoring video image, and inspection record, are displayed on the display at hand 10. Here, the items in the menu 22 is arranged from the top to the bottom in an order based on frequency of selection by the operator in charge of operation. On the other hand, the menu 42 is displayed by pointing out any one of the symbols in the system diagram 5 by the inspector. The items in the menu 42 are the same as these of the menu 22, but the items are arranged from the top to the bottom in an order based on frequency of selection by the inspector, that is, an order of monitoring video image, inspection record, and data setting.
In a corresponding table 130 of events/executing process which is controlled per every display object shown in FIG. 16, the event can be divided into three categories such as kind of operation, button number, and person in charge. The kind of operation includes the following items, and designates a kind of event.
FIG. 17 illustrates a format 131 for designating an output destination. Each of the bits in the format 131 corresponds to a respective charged task. A bit corresponding to a person in charge to receive the output is designated as “1”, and a bit corresponding to a person in charge not to receive the output is designated as “0”. For example, when an output must be transmitted to both an operator and an inspector, the second bit and the third bit in the format 131 are designated as “1”, and other bits are “0”.
If no displayed object exists at the event generated location (step 143), the operation returns to the step 140 and continues to process the next input event. If any displayed object exists at the event generated location (step 143), the operation goes to step 144. In step 144, the input event items in the corresponding table 130 of the event/executing process for the displayed object which is searched in step 142 is examined whether any input items are matched with kind of operation, button number, and person in charge of the input event. If there are any event items matched with the input event (step 145), an output destination of corresponding executing process items is taken out, the table 121 (FIG. 10) is searched using the charged task stored in the output destination as a key, an output device ID is taken out, and the output device ID is set as for an output destination at routine execution (step 146). Subsequently, the routine stored in the routine items in the executing process items is executed (step 147).
The explanation is performed taking the display at hand 10 as an example, but cases of the other displays at hand 30, 50 are entirely the same. In FIG. 19, H is the number of pixels in a horizontal direction of the large display 1, V is the number of pixels in a vertical direction, h is the number of pixels in a horizontal direction of the display at hand 10, v is the number of pixels in a vertical direction, q is one pixel or a several pixels.
0≦curx<h, 0≦cury≦v (2)
Referring to FIG. 20, a process flow of a method for realizing pointing by the mouse 12 on the display at hand 10 and the large display 1 is explained hereinafter. At the start of processing, an initial setting is q<cury<v, and the pointer is displayed at a position (curx, cury) on the display at hand 10 (step 162). As far as cury>q (step 160), when any event such as pressing a button of the mouse 12 occurs, the event processing is executed to the displayed object on the display at hand 10 (step 163). When the mouse 12 is moved forward and cury becomes less than q, that is curY<q, the pointer transfers on the large display 1. That means, a value of cury is set as hV/H (step 164), and the pointer is displayed (step 167) at a position (curX, cury)=(HcurX/h, Hcury/h−1) (step 166). As far as cury<hV/H (step 165), when any event such as pressing a button of the mouse 12, occurs, the event processing is executed to the displayed object on the large display 1 (step 168).
In the above embodiment, a conventional display apparatus is used as for the display at hand 10. However, a see-through display apparatus can be used as for the display at hand 10. The see-through display is a translucent display, and information displayed on it is visible with a background of the display in a superimposed manner. As one example of such see-through display apparatus, there is a see-through head-mounted display described in a reference, November, (1991) ACM Press pp. 9-17.
Furthermore, a see-through display and a conventional display can be used together as for displays at hand. That means, information which is desired to be displayed in a superimposed manner with information displayed on the large display 1000, such as a pointer for designating a position on the large display 1000, and a menu for operating a displayed object on the large display 1000, are displayed on the see-through display, and other information which is not required to be seen in a superimposed manner with the displayed object on the large display 1000 may be displayed on the conventional display.
Advantages of using the see-through display for a display at hand are as follows;
(1) Interference between operators can be completely eliminated. Although displaying a pointer or a menu directly on the large display distracts other operators, displaying the pointer or the menu on an operator's own display at hand does not interfere with other operators' work because the display on his own see-through display is not visible to other operators. For instance, when many pointers are displayed on the large display simultaneously, it becomes difficult to identify one specified operator's own pointer among many pointers, and it causes a problem to be solved. However, if a pointer for each operator is displayed only on his own see-through display, the above problem can be eliminated because each operator sees only his own pointer at any time.
(2) Information displayed on the display at hand and information displayed on the large display can be integrated visually. When a conventional display is used as the display at hand, it is necessary to move a line of sight in order to refer both the information displayed on the large display and the information displayed on the display at hand, and it is difficult tc see both of the above information simultaneously. On the contrary, information displayed on the see-through display is visible with information displayed on the large display in a superimposed manner, and both of the above information can be referred to simultaneously. Furthermore, related information can be displayed next to each other.
1. An interactive control system with plural displays comprising
a plurality of first interactive control apparatuses having input means and output means,
a second interactive control apparatus having a display means,
a see-through display apparatus being used as one of the output means of said first interactive control apparatus, through which see-through display the display of the second interactive control apparatus is visible, and
a position pointing means for designating a position on a display of the display means of said second interactive control apparatus using a pointer which is displayed on the see-through display of the see-through display apparatus of said first interactive control apparatus, said position being determined by the position where said pointer is visible on the display of the display means of said second interactive control apparatus in a superimposed manner.
US09619647 1993-04-28 2000-07-19 Interactive control system having plural displays, and a method thereof Expired - Fee Related US6441802B1 (en)
JP5-102159 1993-04-28
JP10215993A JPH06314181A (en) 1993-04-28 1993-04-28 Interactive control system by plural display and control method therefor
US23036994 true 1994-04-20 1994-04-20
US08969313 US5969697A (en) 1993-04-28 1997-11-13 Interactive control system having plural displays, and a method thereof
US09374263 US6100857A (en) 1993-04-28 1999-08-16 Interactive control system having plural displays, and a method thereof
US09619647 US6441802B1 (en) 1993-04-28 2000-07-19 Interactive control system having plural displays, and a method thereof
US10192726 US20020171628A1 (en) 1993-04-28 2002-07-11 Interactive control system having plural displays, and a method thereof
US10692808 US7057602B2 (en) 1993-04-28 2003-10-27 Interactive control system having plural displays, and a method thereof
US09374263 Continuation US6100857A (en) 1993-04-28 1999-08-16 Interactive control system having plural displays, and a method thereof
US10192726 Continuation US20020171628A1 (en) 1993-04-28 2002-07-11 Interactive control system having plural displays, and a method thereof
US6441802B1 true US6441802B1 (en) 2002-08-27
ID=14319953
US08969313 Expired - Fee Related US5969697A (en) 1993-04-28 1997-11-13 Interactive control system having plural displays, and a method thereof
US09374263 Expired - Fee Related US6100857A (en) 1993-04-28 1999-08-16 Interactive control system having plural displays, and a method thereof
US09619647 Expired - Fee Related US6441802B1 (en) 1993-04-28 2000-07-19 Interactive control system having plural displays, and a method thereof
US10192726 Abandoned US20020171628A1 (en) 1993-04-28 2002-07-11 Interactive control system having plural displays, and a method thereof
US10692808 Expired - Fee Related US7057602B2 (en) 1993-04-28 2003-10-27 Interactive control system having plural displays, and a method thereof
US (5) US5969697A (en)
JP (1) JPH06314181A (en)
EP1929396A2 (en) * 2005-09-19 2008-06-11 Helmut Schröder Computer mouse, computer mouse system and method for controlling said mouse
JP5765999B2 (en) * 2011-04-21 2015-08-19 三菱電機株式会社 Monitoring and control system
US20130146084A1 (en) * 2011-12-07 2013-06-13 Caterpillar Inc. System and method for removing objects from surfaces
JP2016009424A (en) * 2014-06-26 2016-01-18 Kddi株式会社 Operation assist device, operation assist method, and operation assist program
JPS62249276A (en) 1986-04-23 1987-10-30 Hitachi Ltd Individual certifying system
JPS63308681A (en) 1987-06-10 1988-12-16 Toshiba Corp Personal identifying device
JPH01103759A (en) 1987-10-16 1989-04-20 Nec Corp Password detecting device
JPH0239307A (en) 1988-07-29 1990-02-08 Toshiba Corp Plant monitor device
JPH02273857A (en) 1989-04-14 1990-11-08 Fujitsu Ltd Computer system utilizing multimedia
JPH0375693A (en) 1989-08-18 1991-03-29 Hitachi Ltd Method and device for displaying screen
JPH03156557A (en) 1989-08-08 1991-07-04 Mitsubishi Electric Corp Computer system
JPH04218293A (en) 1990-12-18 1992-08-07 Fuji Xerox Co Ltd Multiple color display electroluminescence display
JPH04314153A (en) 1991-04-12 1992-11-05 Toshiba Corp Informaiton liaison device for plant control room
JPH04335698A (en) 1991-05-13 1992-11-24 Yokogawa Electric Corp Process monitoring device
JPH0546559A (en) 1991-08-13 1993-02-26 Fuji Xerox Co Ltd Joint work information processor
JPS54121379A (en) 1978-03-13 1979-09-20 Hitachi Ltd Method of monitoring hierarchy process state
DE3709400A1 (en) 1987-03-21 1988-09-29 Subklew Gmbh control room
JP2539502B2 (en) 1988-12-09 1996-10-02 株式会社日立製作所 Driving support system
JPH0345991A (en) 1989-07-14 1991-02-27 Hitachi Ltd Picture display method and computer system using there for
JPH0371187A (en) 1989-08-11 1991-03-26 Toshiba Corp Plant state supervisory and supporting device
WO1991006960A1 (en) 1989-11-02 1991-05-16 Combustion Engineering, Inc. Advanced nuclear plant control complex
JPH04491A (en) * 1990-04-18 1992-01-06 Toshiba Corp Information display device
E. Bier et al., "MMM: A User Interface Architecture for Shared Editors on a Single Screen," UIST'91, Hilton Head, SC, Nov. 11-13, 1991, pp 79-86.
English Translation of "Notification of Reason(s) for Refusal", Reference No. 1193017161; Dispatch No. 506306, Dispatch Date Sep./25/Heisei 13 Nen (2001).
M. Elwart-Keys et al., "User Interface Requirements for Face to Face Groupware", CHI '90 Proceedings, Apr., 1990, pp 295-301.
M. Stefik et al., ACM Transactions on Office Information Systems, 5:2, Apr. 1987, pp 147-167.
S. Feiner et al., "Hybrid User Interfaces: Breeding Virtually Bigger Interfaces for Physically Smaller Computers," UIST'91, Hilton Head, DC, Nov. 11-13, 1991, pp 9-17.
US20020171628A1 (en) 2002-11-21 application
US5969697A (en) 1999-10-19 grant
JPH06314181A (en) 1994-11-08 application
US7057602B2 (en) 2006-06-06 grant
US6100857A (en) 2000-08-08 grant
US20040085257A1 (en) 2004-05-06 application