Inside a Microsoft Design Experiment
By: Cara Tyler
Tags: Hololens, Fluent Design System, Fluent Design, Mixed Reality, Virtual Reality

Inside a Microsoft Design Experiment

Bringing dimensionality to everyday applications across devices

Dimensionality. It’s a design concept that’s near and dear to the Microsoft Design team’s heart, especially as we continue to evolve our Fluent Design System. It used to be that dimensionality — that immersive, interactive feeling — was expected in gaming, end of story. But now we’re asking: how do you embrace that experience in everyday applications, engaging people on any device? A small team of our designers, technical artists, and app developers used the Universal Windows Platform and the humble weather app to answer that question.

Spoiler alert: it was a challenge. It continues to be a living experiment in the realms of scalability and transcendence: we’re learning, tailoring, fine-tuning, failing, triumphing, and learning again. Here’s a glimpse into what it means to build across dimensions.

Building holographic

We chose HoloLens as our starting line, building a 3D experience in mixed reality that would be easy for newcomers to grasp and enjoy. From there we’d figure out how to bring that experience into a 2D application. The whole idea being: let’s invite people in; use the same components across experiences to build familiarity and responsiveness and a sense of place.

Why a weather app? Turns out you can have a lot of fun with it and build something super engaging in a short timeframe. Plus it’s based in reality, knowable to everyone, and gave us the chance to research some cool places. We focused our attention on phone, PC/tablet, and HoloLens, and flew from there, completing work in a few weeks that explores different ways to bring dimensionality and delight into a universal experience.

The project kicked off in May, a damp and chilly time here in the Seattle area (shocking), so we had sunny, coastal locations on the brain. We experimented a bit to find the right scale and level of detail to build four atmospheric dioramas in HoloLens: Tuscany, Mykonos, Cape Cod, and Monument Valley. The experience opens on the edges of each scene: a slice of a miniature world dropped into your everyday environment. From there, you’re invited to interact with the objects — becoming true scenes that adapt as you move closer and walk around them. Our operative theme here is engagement. You start at the level of information architecture: what is the temperature in Tuscany? Then, we invite you in. Come get closer to the terrain. Now you see the light and the atmosphere. Next, the detailed motion and spatial sounds. Now, narrative details. Someone waving at you, trees swaying.

A helicopter hovers in Monument Valley

Because you’re in a mixed reality HoloLens, it’s possible someone in your real world will walk through your hologram. And you will absolutely scream no! as it happens because now, you’re living this environment. It happens fast: you’re feeling something, you’re in it, and you’re emotionally connected. It’s sort of like getting lost in a snow globe. A dynamic, mesmerizing place within a place.

To get the right size, scale and placement of objects, we relied on our 3D artist to work closely with the designers in Maya (3D animation software). This is where we did the blocking to determine the right size, scale, and placement of the objects within a larger scene, just like directing a movie. Each element needed to be sized just so to maintain realistic proportions regardless of viewing distance or angle.

Concept sketch for Cape Cod

Early blocking of Cape Cod scene to determine scale of elements

We got the best results — and moved quickly — when we baked shadows into the textures. The realism of each scene came from careful attention to detail and an artful eye. For example, the distinct size, direction, and movement of the waves helped differentiate the Atlantic coast from an island in the Aegean Sea (details matter!). The transition between dioramas is relatively fast and simple, keeping your focus on the scene and the feeling it brings, not the motion itself.

Final art ready for lighting and animations

Cape Cod diorama integrated into the experience

What else makes an appearance? Sunbathers. A sailor. Seagulls, cows, a tractor. We had a lot of fun telling the story and inviting people in. We added subtle animations and learned that polygon count, texture size, and the number of controlling joints all impacted the animation’s smoothness. Ambient sounds (crashing waves) and distinct sounds (moo!) were the final touches to bring everything to life.

Simple, minimally animated seagull and sunbathers in the Mykonos scene.

Scaling it to 2D

So we created a pretty, fun, interactive thing for HoloLens. Now, how to scale that back to phone and tablet, and make 2D and 3D content that co-exists naturally? Could we reuse the hologram within a flat UI and still make it feel engaging? We’re fortunate that the Universal Windows Platform makes this easy — check out this detailed how-to in another sample weather app, Atmosphere.

One-step conversion is still a dream, and we had to do some manual tailoring to make the diorama look great on 2D devices. We adjusted the shaders for the lighthouse beam to retain a realistic look, and turned off the MIP map compression for the diorama textures. We also experimented with colors that matched the diorama before landing on a grayscale background.

Cape Cod diorama optimized for the 2D app

Adaptive input

The Fluent Design System is all about translation and cohesion. Our goal in this experiment was to make each element translate in a way that’s appropriate for the type of device and primary input (touch, mouse, gaze) being used. We designed to maintain the continuity of the experience from both visual and interactive perspectives, while making sure things worked naturally and familiarly on each device. We wanted the HoloLens experience to stay true to holographic and 3D form, but still be approachable and easy for new users — making gesture, gaze, and voice as familiar as using a PC. The 2D experience needed to optimize for the available screen real estate and feel natural with touch and mouse interactions, while still feeling as immersive as mixed reality. Designing outside the box, within a box, as it were.

Here’s a glimpse into some of the work! Take note of the button to switch between scenes — a common design element for all three form factors to help with wayfinding.