情報処理学会 インタラクション 2023
IPSJ Interaction 2023

2A-112023/3/1

A 3D Window Management System
Using Viewpoint Manipulation
Rin Ohsugi1

Yuki Yamashita2

Miki Katsuragi2

Yuji Sasaki3

Abstract: In window management systems of Windows and macOS, simultaneous display of multiple windows andswitching displayed windows are implemented as separate operations. In this paper, we propose Air Manager, a window
management system that can perform both functions with a single operation using a viewpoint shift. Through this system,
we have devised an operation method combining the simultaneous display of multiple objects and switching displayed
objects, which are the two separate operations in conventional window management systems. We found a possibility tosolve the problem of window management systems of Windows and macOS that require a separate operation each time tochange the way windows are displayed between single-tasking and multi-tasking.

1. Introduction
As the performance of personal computers has improved, it has
become common for users to launch many applications
simultaneously and work with them, either displaying themsimultaneously or moving back and forth between them. Whenusers launch many applications, there are several ways to display
and control the multiple windows they use to work with them.
Each of these operations is based on one of the following twoconcepts. One is to display multiple objects at the same time. The
other is to switch objects to be displayed.
The tiled window method is a conventional display method in
window management systems that employs the concept ofsimultaneous display and can display multiple windows
simultaneously. In this tiled window method, windows are laidout on the desktop with no overlap. Thus, users can browse
multiple windows simultaneously. The overlapping window
method employs the concept of switching to enable changing thewindows to be displayed. In this overlapping window method,windows can be arranged with overlapping on a desktop. This
allows more windows to be placed on a desktop. Windows hiddenin the back can be brought to the front by clicking on them, andthe windows displayed on the screen can be switched. In thismethod, the size and position of the windows can also be adjusted.
Thus, it is possible to arrange multiple windows without
overlapping each other and display them simultaneously.
However, it requires many operations since window switching
and window size and position adjustment are two differentoperations. Comparing the tiled window method and theoverlapping window method, it has been demonstrated that thetiled window method works more efficiently in some situations
due to the difference in the number of operations [1].
The concept of switching is also applied to desktops. Virtual
desktops are designed to duplicate desktop screens to expand theworkspace to switch tasks between them.
1 N High School
2 The University of Tokyo
3 Keio University© 2023 Information Processing Society of JapanIn Windows and macOS, operation methods based on theconcept of simultaneous display and those based on the conceptof switching are implemented separately.
For example,switching between single windows on the desktop is mainly
performed using the overlapping window method, while
simultaneous display of multiple windows during multi-tasking isperformed using the tiled window method [2][3]. Virtual
desktops are implemented to switch multiple windows together[4][5]. However, since the operation method for simultaneously
displaying windows and for switching between displayed
windows are implemented differently, additional steps are
required to switch them. For example, suppose that whenchanging windows on the desktop and using several windows,
users desire to refer to them simultaneously instead of switching
between them. In this case, a separate process of dividing thescreen or adjusting the window size for simultaneous reference
would be necessary. Thus, when users want to work with multiple
windows, they have to choose either simultaneous display orswitching. At this point, users must switch between different
operations each time.
This paper proposes a new concept in window management
systems that eliminates the need to select whether to display
windows simultaneously or to switch between them. The concept
is viewpoint moving, which combines displaying multiple objects
simultaneously and switching what is displayed. In this paper, we
propose a window management system, Air Manager, that places
all windows in a single desktop space and enables both to display
simultaneously and to switch them by a single viewpoint control.

2. Previous WorkA number of previous research on window management systemsinvolved approaches to allow several windows to be displayed
simultaneously while allowing them to be switched collectively.
There are several methods to achieve this. One of them is to
apply the idea of switching to desktops, whereby desktops are
duplicated and can be switched between them. The other isallowing multiple windows to be switched once within a desktop.

498

情報処理学会 インタラクション 2023
IPSJ Interaction 2023

Rooms [6] and Task Gallery [7] are the systems that employ thefirst method. Rooms enable users to create multiple desktops andswitch between them. This system can switch between several
windows at once, grouped by desktops. Thus, users can switch
between screens with windows displayed simultaneously byswitching between desktops with multiple windows arranged
without overlap. Task Gallery enables switching between tasks
through a three-dimensional space that uses a gallery as ametaphor. Users can duplicate desktops and place them on walls
or other locations within the space, making it easier to managetasks through the user's ability to recognize space. In addition, byplacing frequently used applications, in easily accessible
locations, windows can be easily moved onto the desktop,facilitating the transition to multitasking.
Stage Manager [8] and Docking Window Framework [9] are thesystems that employ the second method. Stage Manager is awindow management system implemented in macOS. It enables
users to place a group of windows as tasks in a thumbnail view at
the left edge of the screen, and by clicking on the thumbnail,users can switch windows by task. This allows multiple windows
to be switched with the same click operation as a single window.
The Docking Window Framework allows multiple windows to be
moved together by docking them. This allows several windows to
be operated as a single window within the desktop, and multiple
windows can be switched together.
These window management systems enable task switching by
allowing multiple windows to be operated together. However,none of them integrates the window switching operation with theoperation required to display multiple windows simultaneously.
For example, in Rooms and Task Gallery, there are operations toplace specific windows on the desktop and operations to switch
between desktops. Stage Manager and Docking Window
Framework also require two operations: one to select windows
used for multitasking and the other to switch between them. On
the other hand, Air Manager can basically shift fromsingle-tasking to multi-tasking with only a viewpoint operation.

3. Proposal of Air Manager
3.1 Concepts
In Air Manager, as the viewpoint can be moved, several
windows can be displayed simultaneously and switched between
them by simply moving the viewpoint. Thus, the simultaneous
display of multiple windows and switching between windows tobe displayed are not viewed as separate concepts in Air Manager
and can be performed with a single operation. Air Manager ischaracterized by allowing users to arrange windows in the x- andz-axis directions within a single continuous three-dimensional
space, and to switch between these windows by viewpoint. In
addition, the viewpoint can be moved along the x- and z-axes,
allowing windows to be positioned off-screen and expanding theworkspace on a single desktop.

© 2023 Information Processing Society of Japan

2A-11
2023/3/1

Air Manager's viewpoint operation has the role of threefollowing functions in a conventional window management
system.
l
Switching between front and rear windows in an
overlapping window system.
l
Virtual desktops for task switching.
l
Screen splitting for multi-tasking.
In Air Manager, to effectively utilize the above-mentioned
viewpoint operation features, arranging windows in the following
manner during setup is desirable.
(1) Group the windows related to the work as tasks and alignthem in the x-axis direction with the same z-axis coordinates asshown in Figure 1. (In Figure 1, window A, B, and C are grouped
as one task and window D, E, and F are grouped as another task.)(2) Place the different tasks in a row by shifting them in thez-axis direction as shown in Figure 1. (In Figure 1, window D, E,and F are placed behind window A, B, and C.)
By doing so, switching between single windows, task switching(switching between multiple windows), and multitasking canbasically be performed only by operating the viewpoint.

Figure 1. preferable placement of windows in Air Manager
Due to these points, for example, Air Manager does not switch
operation methods in the following situations: switching between
two overlapping windows on the desktop and users occasionallydesire to refer to both windows at the same time. In this case,window management systems in Windows and macOS require atransition to the split-screen or window resizing operation. Bothoperations require a separate operation different from thewindow-switching operation on a desktop. Air Manager, on theother hand, does not require such operations. Two windows linedup on the left and right can be switched to the left window bymoving the viewpoint to the left (Figure 2) or to the right windowby moving the viewpoint to the right (Figure 3). In this case, bymoving the viewpoint to a position between the two windows,
both can be displayed at the same time. (Figure 4) Therefore, Air
Manager has the advantage of being able to simultaneously
display multiple windows and switch between them with a single
operation. However, some exceptions exist. For example, if
windows that users want to display at the same time are notadjacent to each other on the left or right, it is necessary to
relocate those windows.

499

情報処理学会 インタラクション 2023
IPSJ Interaction 2023

2A-11
2023/3/1

other hand, when clicking on another window, these gestures aredeactivated. This allows the user to use the same gestures as
when normally operating a window. Similarly, windows haveactive and inactive states, and once a window is clicked, it
becomes active. A window in the active state moves in thexy-plane to the direction along which the cursor moves and canmove along the z-axis at regular intervals in the same way as theviewpoint can be scrolled. These functions allow users to placewindows in different positions, including depth, and to switch
tasks by viewpoint operation without changing their placement.

4. Development
Figure 2. View of the screen when the viewpoint is moved to theleft to match the left window

Figure 3. View of the screen when the viewpoint is moved to theright with the window on the right

Figure 4. View of the screen when the viewpoint is moved to the
nearby area between the two windows to display both windows
simultaneously.
3.2 Operation Method
The desktop screen of the proposed system is a projection of a
3D space from a single viewpoint. The viewpoint can be
manipulated by dragging and scrolling the ground object
displayed at the bottom of the screen. By scrolling, the viewpoint
can be moved in the z-axis direction at regular intervals
according to the direction of the wheel or trackpad. The groundobject has two states: active and inactive. Once clicked, it
becomes active and drag and scroll gestures are activated. On the© 2023 Information Processing Society of JapanWe developed a prototype using Unity to validate the proposed
method. This system runs on macOS and can handle all windows
displayed on the desktop. The implementation method is divided
into the following two modules: (1) Window Setup Module todisplay windows, (2) Objects Control Module to move theviewpoint and windows.
4.1 Window Setup Module
Windows in this system are projected onto a game object inUnity, which is captured on the desktop screen. For capturingwindows, we used mcDesktopCapture [10], which is based on
ScreenCaptureKit. Selecting an application from a dropdown
button at the top of the screen generates a new window object forthat window. The generated window object is attached with atexture of the captured image of the application window selectedin the dropdown and displayed. The object's scale values (x- andy-axis scales) are calculated by dividing the number of pixels of
the captured window by 1000 times the size of the object's mesh(a constant based on the ratio of the number of pixels to objects inUnity). In other words, the size and screen ratio of the window
object in this system is based on the resolution according to thewindow size on the macOS desktop.
4.2 Objects Control ModuleThis section describes the behavior related to object movement.
The screen of this system shows the view from a camera in Unity,
which is an object used to capture the world (3D space in thiscase) to be displayed to users. A ray is sent from the center of thecamera's viewpoint in the direction of the mouse cursor, and thesystem identifies objects that are hit by the ray. This allows themouse cursor to click on a window in the 3D space. The click isdetermined by the Collider, which is assigned to the window andground objects. Once the window object detects a click, thecoordinates of the window object can be manipulated withgestures. It can move in the x and y-axis directions in response tothe up/down and left/right movements read from the mouseposition information while users are clicking. (Figure 5) Inaddition, the coordinates of the z-axis change with the number oftimes the object is scrolled; one scrolling movement moves theobject by 3 in the coordinates in Unity. (Figure 6) (If the value ofthe scrolling rotation is positive and the scrolling is in thebackward direction, the value is +3; if the value of the scrolling500

情報処理学会 インタラクション 2023
IPSJ Interaction 2023

rotation is negative and the scrolling is in the forward direction,the value is -3). When Ray hits an object on the ground, thecoordinates of the camera can be manipulated with gestures. The
left/right movement is read based on the mouse position while
clicking, and the viewpoint moves in the opposite direction ofthat movement. (Figure 5) This makes it possible to move theviewpoint by dragging objects against the ground. The camera
can also be moved in the z-axis direction by clicking on theground object followed by scrolling, in the same manner as thewindow. (Figure 6)

2A-11
2023/3/1

window management systems that require a separate operationeach time when changing the window display between
single-tasking and multitasking. Further validation experiments
are necessary to evaluate the usability of Air Manager in thefuture. We also plan to solve problems in Air Manager, such as
the problem where users have to move windows when they wantto simultaneously display two windows that are not next to eachother on the left or right.
Acknowledgments This work is supported by KADOKAWADWANGO educational institute, Research Club.

Reference[1]

Figure 5. The top image shows the positioning of objects in 3Dspace.

Bly, B. and Rosenberg J.: A comparison of tiled and overlapping
windows, In Proc. CHI `86, 1986, pp.101-106.
[2] “Snap your windows – Microsoft Support”
https://support.microsoft.com/en-us/windows/snap-your-windows-8
85a9b1e-a983-a3b1-16cd-c531795e6241(accessed 2022-12-10)[3] “Use two Mac apps side by side in Split View – Apple Support”
https://support.apple.com/en-us/HT204948(accessed 2022-12-10)[4] “Meet Windows 11: Basics – Microsoft Support”
https://support.microsoft.com/en-us/windows/meet-windows-11-thebasics-a7519756-6807-41e4-be66-ed3b2c0abe0d(accessed 2022-12-10)[5] “Work in multiple spaces on Mac – Apple Support”
https://support.apple.com/guide/mac-help/work-in-multiple-spacesmh14112/mac(accessed 2022-12-10)[6] Henderson, A., Card, S. K., “Rooms: The use of multiple virtualworkspaces to reduce space contention in a window-based graphicaluser interface”, ACM Transactions on Graphics, 1986, p.212-242[7] Robertson, G., Dantzich, M.v., Robbins, D., Czerwinski, M.,Hinckley, K., Risden, K., Thiel, D., Gorokhovsky, V.: The taskgallery: a 3D window manager; In Proc. of CHI ’00, 2000,p.494-501.
[8] “Organize your Mac Desktop with Stage Manager”
https://support.apple.com/guide/mac-help/use-stage-manager-mchl534ba392/mac(accessed 2022-12-10)[9] Hirohito Shibata, Kengo Omura. Proposal and evaluation of awindow system that enables constructing workspaces, Transactionsof the Japanese Society for Artificial Intelligence, 2011, p.237-247[10] “Github fuziki/mcDesktopCapture”
https://github.com/fuziki/mcDesktopCapture(accessed 2022-11-20)

Figure 6. The image shows the positioning of objects in 3Dspace.

5. Conclusion
This paper proposes Air Manager, a window management
system that enables both simultaneous display of multiple
windows and switching between them using simple viewpoint
operations. Through this system, we have devised an operation
method combining the simultaneous display of multiple objects
and the switching of displayed objects, two separate operations inconventional window management systems. We found thepossibility of solving the problem of Windows and macOS© 2023 Information Processing Society of Japan501