JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

1

TimeTubesX: A Query-Driven Visual Explorationof Observable, Photometric, and PolarimetricBehaviors of Blazars
Naoko Sawada, Student Member, IEEE, Makoto Uemura, Johanna Beyer,
Hanspeter Pfister, Senior Member, IEEE, and Issei Fujishiro, Member, IEEE
Abstract—Blazars are celestial bodies of high interest to astronomers. In particular, through the analysis of photometric andpolarimetric observations of blazars, astronomers aim to understand the physics of the blazar’s relativistic jet. However, it is challenging
to recognize correlations and time variations of the observed polarization, intensity, and color of the emitted light. In our prior study, we
proposed TimeTubes to visualize a blazar dataset as a 3D volumetric tube. In this paper, we build primarily on the TimeTubes
representation of blazar datasets to present a new visual analytics environment named TimeTubesX, into which we have integrated
sophisticated feature and pattern detection techniques for effective location of observable and recurring time variation patterns inlong-term, multi-dimensional datasets. Automatic feature extraction detects time intervals corresponding to well-known blazar
behaviors. Dynamic visual querying allows users to search long-term observations for time intervals similar to a time interval of interest(query-by-example) or a sketch of temporal patterns (query-by-sketch). Users are also allowed to build up another visual query guided
by the time interval of interest found in the previous process and refine the results. We demonstrate how TimeTubesX has been usedsuccessfully by domain experts for the detailed analysis of blazar datasets and report on the results.
Index Terms—Visual analytics, feature extraction, visual query, multi-dimensional, time-dependent visualization, astrophysics, blazar

F

1

I NTRODUCTIONB

LAZARS are important to the studies of astronomy and highenergy astrophysics. They are in a class of extremely brightgalactic nuclei called active galactic nuclei (AGN) that shoot out
a stream of particles from their center [1]. A relativistic jet isangled directly toward the Earth from a central black hole ofa blazar (see Fig. 1). However, relatively little is known about
the intricate physics of these jets’ structures. Astronomers pay
careful attention to blazars because the jet radiation of blazars
is more amplified than that of other AGN. To demystify thejets’ structures, astronomers need to analyze correlations andtime variations in photometric and polarimetric observations of
the emitted light. For instance, a light burst (i.e., flare) is oneof the most distinctive behaviors of blazars [2], [3]. Identifying
flares requires astronomers to scrutinize correlations of the emitted
light’s intensity with polarization and/or color, because looking at
the time variation of the intensity alone is insufficient. In addition
to flares, some astronomers are interested in the polarization
direction of light and whether it rotates [4], [5]. It is a controversial
question in astronomy whether such rotations are real or due torandom variations of polarization. To verify polarization rotations,correlations of the polarization with the intensity and/or the color
are crucial. Moreover, astronomers would like to locate time intervals with common features in time variations to understand what

•
•
•
•

N. Sawada is with Keio University.
E-mail: naoko.sawada@fj.ics.keio.ac.jp.
M. Uemura is with Hiroshima University.
E-mail: uemuram@hiroshima-u.ac.jp.
J. Beyer and H. Pfister are with Harvard University.
E-mail: {jbeyer, pfister}@seas.harvard.edu.
I. Fujishiro is with Keio University and Hangzhou Dianzi University.
E-mail: fuji@ics.keio.ac.jp.

Manuscript received May 9, 2020: revised August 26, 2020.

happens inside the jet. However, it is an overwhelming process forthem to manually examine all time intervals in long-term, multidimensional datasets in order to detect these significant patterns
and validate hypotheses.
To support these exploration challenges, we have developed
TimeTubesX, a new visual analytics environment for blazar observations. This novel system extends our previous visualization
scheme, called TimeTubes [6], which allows users to interactively
explore multi-dimensional, time-dependent observation datasetsof blazars in a unique 3D tube view and to combine datasetsfrom multiple observatories into a single visualization session
termed visual data fusion. In TimeTubesX, we place more focus
on visual analysis and automated and semi-automated approaches
for feature and pattern detection. The automatic feature extraction
methods, which are in part an extension of our previous work [7],detect time intervals corresponding to well-known blazar behaviors, such as flares and rotations. The dynamic visual querying
methods, on the other hand, are useful for analysis and detectionof recurring patterns that have not yet been explored in detail.
Our first contribution is a detailed goal and task analysis
performed with domain experts to design a visual analytics framework for blazar observations. Subsequently, we have designed
and implemented TimeTubesX as a web-based and open-source
system. Our second contribution lies in powerful automatic feature
extraction methods for well-known blazar behaviors and an intuitive visual query platform for multi-dimensional, time-dependentdata. Finally, as our third contribution, we demonstrate the usefulness of TimeTubesX through experiments with a synthetic dataset,real data analyses, and feedback from domain experts. We envisionthat our visual query mechanisms could be applied to multidimensional, time-series datasets in other domains to help users
identify recurring patterns or similar time intervals.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

2

R ELATED WORK

Our target blazar datasets are multi-dimensional (mD), semistructured, and time-dependent. This section reviews prior workin astronomical visualization and visual query systems for mD
and time-series data.
2.1

Visualization for Astronomical Data

Most astronomers work on analyzing signals from distant celestial bodies measured from the Earth [8]. ESASky [9] andOpenSpace [10] provide views for parts of the sky that are integrated from multiple data sources. Almryde et al. [11] visualize
the temporal evolution of the position, mass, and radial velocity
of dark matter halos. Li et al. [12] realize simple visual analysis across multi-wavelength datasets. IGM-Vis [13] is a visual
analytics tool for quasar sightline data. Some approaches [12],[13], [14], [15], [16] provide linked-view visualization systems
that allow users to explore multiple types of visualizations fordeeper analyses. Many techniques have been developed for the
analysis of astronomical data, but typically, blazar researchers
have used animated scatterplots (see Section 4.1) because there
were no other effective techniques for blazar observations prior to
our early attempt, TimeTubes [6].
2.2Visual Queries

Visual query systems allow users to intuitively search for patternsof interest in their data. Query-by-example aims to find data
samples that are similar to a user-specified example, whereas
query-by-sketch allows users to directly draw the shape of interest.
Query-by-example. Hochheiser and Shneiderman [17] propose TimeSearcher, a visual exploration tool for time-series data.
Users can select portions of timelines by using rectangular widgets [18], [19]. Holz and Feiner [20] propose a relaxed selection
technique for time intervals through which users define the levelof similarity either by the input speed or by spatial deviation froman original timeline. The idea of detecting the input speed was
adopted for our sketching interface. However, these interactions
are geared only toward univariate time-series data and are notappropriate for our mD blazar data.
Query-by-example methods for mD data have also been proposed. Martin and Ward [21] describe brushing techniques forvisualizations, such as scatterplots matrices and parallel coordinateplots. Elmqvist et al. propose interactions for constructing visualqueries with star plots [22] and multiple scatterplots [23]. Scribble
query [24] allows users to form visual queries by scribblingaxes of parallel coordinate plots. Several works have focused on
unstructured mD datasets [21], [22], [24], but they are not versatile
enough to examine correlations among our idiosyncratic variables(see Section 3.1), and they do not support queries for structured
time-series data. The scatterplots by Martin and Ward [21] andElmqvist et al. [23] can be partially useful for structured data, butthey cannot be applied to time-series data.
Query-by-sketch. Early examples of query-by-sketch tools
include QuerySketch [25] for temporal databases and QueryLines [26] for multiple timelines. TimeSketch [27] supports penand touch-based query specification but focuses only on univariate time-series data. To manage the ambiguities of hand-drawn
sketches, several systems discretize the query and datasets into
symbolic or quantized representations [28], [29]. Alternatively,
Correll and Gleicher [30] define invariants for sketches. Qetch [31]2

provides a matching algorithm that takes human sketching errors into account. Our query-by-sketch interface uses sketchinginteractions similar to those of Qetch [31]. Both techniques target
univariate time-series data. Shao et al. [32] introduce sketch-based
queries for large sets of scatterplots, in which the system supports
users with shadow-drawing suggestions. The suggestions allow
users to draw a pattern based on actual values rather than their
imaginations. We use this idea as inspiration for our fact-guided
querying (see Section 6.4). While these systems support querying
structured mD data, they cannot be applied to time-series data.
Several sketch-based techniques for multi-scalar volume data
allow users to select regions of a 3D object through an ordinary 2D
painting tool. The techniques by Owada et al. [33] and Igarashi etal. [34] extract a plausible 3D region from the volume by following
contours or isosurfaces, respectively. Tzeng et al. [35] present an
method for specifying high-dimensional classification functions
for the volume by brushing regions of interest. BrainGazer [36]provides simple interactions for the selection of a region of interestin the volume using drag-and-drop functionality. These techniques
can only target multi-scalar volume data.
Combinations of query-by-example and query-by-sketch.
Query-by-example techniques do not address how to find theinitial data samples of interest, while query-by-sketch techniquesmust surmount the user-introduced ambiguities of sketches. Combining both styles helps to resolve this trade-off, as addressed by
Bernard et al. [37] and Lee et al. [38]. The system formulated by
Bernard et al. [37] supports univariate time-series data, while theone by Lee et al. [38] supports unstructured time-series data. To
get the best of both worlds, TimeTubesX supports both types ofquerying interactions.

3

D OMAIN A NALYSIS

In this section, we specify our blazar observation data and define
domain-specific goals and visualization tasks that were identified
in consultation with our domain experts.
3.1

Blazar Observation Data

Multiple observatories worldwide collect blazar data; however,their observed properties or units of measurement differ dependingon the telescopes being used. To reduce uncertainties due tomissing observational data and observation errors, TimeTubesX
allows users to fuse multiple datasets for the same blazar fromdifferent observatories through the automatic handling of potentially different definitions of observed variables [6]. Thus,
users can, for example, include data from Hiroshima University
and the University of Arizona in the same visualization session.
TimeTubesX is designed to primarily process datasets acquired by
Hiroshima Astrophysical Science Center at Hiroshima University,
which uses a 1.5m Kanata telescope.
The photometric and polarimetric datasets of Hiroshima University include six time-dependent variables: q, u, εq , and εu for theobserved polarization of the light (Fig. 1 (A)), I for the observed
intensity (Fig. 1 (B)), and C for the observed color (Fig. 1 (C)).
Linear polarization is represented by Q and U, the so-called Stokes
parameters. We mainly use fractional values of Q and U whereq = Q/I and u = U/I, because q and u describe the behaviors
of a blazar more effectively than Q and U [39]. We call theplane composed of q and u the Stokes plane. εq and εu act as
the observation errors of q and u, respectively. The polarization
degree (PD) and polarization angle (PA) derived from q and u are

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20203

Obscuring torus𝑦𝐼

𝜖! (𝑡)B
A

Relativistic jet

Black hole𝑧=𝑡𝑧(𝑞(𝑡), 𝑢(𝑡), 𝑡)𝜖" (𝑡)CD(a)

Blazar

2εq

u

B

22εu
PD

PA×2
O (q, u) = (0, 0)

Stokes (q, u) plane
for polarizationq

I: Observed intensity of the lightC

C: Observed color of the light

Fig. 1. An observed blazar and its features. A blazar’s central black holeemits a relativistic jet toward the Earth. The telescope observes its lightin terms of (A) polarization, (B) intensity, and (C) color.

used to describe the optical polarization. PD denotes a distancefrom the origin of the Stokes plane, while PA denotes one half ofthe polar angle in the Stoke plane, as illustrated in Fig. 1 (A).
The observation frequency differs depending on observatories.
The median observation interval of Hiroshima University is once
every two days. Note that the datasets have missing data dueto the rotation and/or revolution of the Earth, bad atmosphericconditions, and so on. In this paper, we process datasets from
Hiroshima University for two blazars: BL Lac and 3C 454.3. The
two datasets for BL Lac contain 285 data samples in each dataset
from May 2008 to December 2011, while the one for 3C 454.3
contains 355 data samples from July 2008 to July 2014.
3.2

A

𝐶

𝑧=𝑡

(b)

𝑧 = 𝑡′
B
O C

𝑧

D

𝑥

O’ C’
D’
𝑥

(c)

Fig. 2. Spatial mapping in the TimeTubes view. (a) Observation values ofpolarization decide the position and shape of an ellipse; (b) observation
values of intensity and color colorize the ellipse with reference to acolormap; and (c) the neighboring ellipses are smoothly connected inchronological order to yield a tube shape.

TelescopeAA’(𝐶 𝑡 , 𝐼(𝑡))

𝑥

B’

𝑦

𝑦

Goals and Visualization Tasks

We interviewed two astronomers at Hiroshima University, one at
Stanford University, and four at Boston University to clarify thedomain goals and visualization tasks. The main objective of theseastronomers is to quickly extract and analyze features in temporalvariations of the observed polarization, intensity, and color thatare occurring at multiple time intervals. TimeTubesX effectively
supports the following four domain goals:
G1–Enhance the reliability of blazar observations. Datasets
always contain missing data and observation errors. Astronomers
need to assess whether what they have found is plausible byanalyzing the length of missing data periods and the size of errors.
G2–Identify flares and rotations. Astronomers typically pay
attention to time intervals with dynamic time variations, suchas flares and rotations. The analysis of time intervals with such
dynamic variations helps astronomers demystify jets’ structures.
G3–Locate recurring blazar behaviors. Besides well-known behaviors such as flares and rotations, recurring patterns or common
features can also exist. Thus, upon finding an interesting patternor feature, astronomers want to locate other time intervals similar
to it.

G4–Explore time intervals validating a hypothesis of blazar
behaviors. Through their analyses or experiences, astronomers
sometimes make a hypothesis (e.g., that the flares of a blazar
tend to co-occur with a specific polarization variation pattern).
Astronomers need to address time intervals that might validate thehypothesis.
To attain these four goals, we have identified five visualization
tasks that TimeTubesX should support:
T1–Uncertainty-aware visualization. Users should be able toperceive the reliability of data through visualizations.
T2–Analysis across datasets. To compensate for missing data, thesystem must allow users to aggregate datasets for the same target
from different sources. Analyses across datasets should also be
supported to address features that are common to multiple targets.
T3–Analysis of dynamic variations. Time intervals with drasticchanges in a short time period and those with unusually large time
variations should be automatically extracted.
T4–Similarity analysis of a specific region/shape of interest and time intervals. Users should be able to identify regions/shapes of interest (ROIs/SOIs) and search for similar time
variation patterns at other time intervals without using complex
query languages or parameter settings.
T5–Fuzzy search for time intervals similar to a specific
ROI/SOI. The system should provide not only exact matches between a ROI/SOI and time intervals but also fuzzy or approximate
matches.
To support G1, our previous work, TimeTubes [6], has alreadysupported T1 and T2. Observation errors are encoded in the appearance of the 3D volumetric tube (T1). An ellipsoidal snapshot
and a white cruciform axis appear at each observation timestamp
to distinguish missing data (T1). Analysis across datasets (T2) is
supported by visual data fusion, which allows users not only toaggregate multiple datasets for the same blazar but also to effectively compare multiple datasets for the same or a different blazar
in a single visualization session. In this paper, we mainly focus on
feature and pattern detection methods to support the remaining
three tasks (T3, T4, T5). Consequently, we have designed an
integrated visual analytics framework for blazar observations that
supports all identified goals (G1–G4) and tasks (T1–T5).

4

S YSTEM D ESIGNIn this section, we provide an overview of visual encoding for
blazar data and the visual exploration framework of TimeTubesX.
4.1

Visual Encoding for Blazar Data

Astronomers have used three animated scatterplots with error bars
to visualize their multi-dimensional, time-dependent observations

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

TimeTubesX

4

Query matches:
overview

Automatic feature extraction
Anomaly
Flare
Rotation

Input

Timestamp

Dynamic visual querying
Query-by-example Query-by-sketch

Timestamp
Comparativeline charts
Distance

Data
etc.
Knowledge

TimeTubes view

Screenshot
Distance

Visual Data
FusionQuery matches:
detailed analysis

Results are sorted by:

Data
Observatory A
Observatory B
Observatory C…

Visualization

Interactive Feature Analytics

Feature and pattern detection

Rotation angle
etc.

Scatterplots view
Light curve

Stokes plane

Fact-guided querying

Astronomer
Fig. 3. The visual exploration framework of TimeTubesX. Users can load multiple datasets into a single exploration session through visual data
fusion. (A) Users specify a query to extract features of interest; (B) query results are sorted by relevance; (C) individual results can be analyzed
in high detail and compared to each other; (D) an extraction result can be re-used as an input for a new visual query; and (E) users can visually
explore the results in the TimeTubes and scatterplots views.

(see the accompanying video): one for time variation of I, termed
light curve, another for the Stokes plane, and a third for thecorrelation between I and C. During the animation, individual
observations are highlighted in red in order of observation.
Instead of animating multiple 2D scatterplots, TimeTubesX
expresses blazar observations as a single 3D volumetric tube. In
the following part, we give an overview of the TimeTubes view,which was originally proposed in our previous work [6]. The
TimeTubes view allows users to see correlations and variationsof variables over time at a glance, as illustrated in Fig. 3 (E). In
the current version, users need to import .csv files with column
names from their local environment into TimeTubesX. We use aleft-handed coordinate system to assign q and u to the x and y axes,
respectively, and time t to the z axis. We encode the polarization
parameters (q, u, εq , εu ) at each timestamp t as an ellipse centeredat the point (x, y, z) = (q(t), u(t),t) with a width of 2εq (t) anda height of 2εu (t), as depicted in Fig. 2 (a). Therefore, the x–ylocation of an ellipse indicates the polarization at a certain timestamp, while the size of the ellipse indicates the uncertainty ofthe measurement. To properly render a 3D tube, we set the value
range of the Stokes plane in the TimeTubes view with reference
to the standard deviations of q and u in the datasets. In the current
TimeTubes view, we empirically map a single day to a single voxel
along the z axis. We colorize the ellipses according to I(t) and C(t)based on a user-defined 2D colormap (Fig. 2 (b)). The TimeTubes
view connects neighboring ellipses in chronological order, using
centripetal Catmull-Rom splines to form a 3D volumetric tube
(Fig. 2 (c)). To further reflect the reliability of the observations,
the TimeTubes view offers an adjustable opacity transfer function.
Multiple concentric tubes with different transparencies (i.e., higher
opacities for inner tubes) compose a single tube that allows
users to intuitively perceive the uncertainties of observations.
Specifically, a time interval with small errors looks like an opaquetube, whereas a time interval with large errors looks more semitransparent and fuzzy.
Compared with the initial animated scatterplots, the TimeTubes view provides more uncertainty-aware visual encoding forthe analysis of blazar behaviors (T1). Astronomers do not needto scrutinize multiple plots to understand correlations between
variables or move sliders back and forth to track time variations.

4.2

Visual Exploration Framework

The design of TimeTubesX supports the visualization tasks outlined in Section 3.2. Fig. 3 illustrates our visual exploration
framework. The user workflow starts with visual data fusion [6]to create a unified dataset for all subsequent analysis steps (T2).
For initial feature and pattern detection (A), users can either rely
on automatic feature extraction methods for well-known blazar
behaviors (T3) or define their own visual queries for a ROI/SOI(T4, T5). The system ranks the results of the feature and patterndetection stage and shows the ranked matches (B). Users cansort the results to, for instance, focus on time intervals that havethe largest rotation or are the most similar to the input pattern.
The detailed analysis (C) helps users understand the behavior at
the extracted time intervals and classify the results. To support
iterative refinement of queries, users can build a follow-up query
based on the result of a query (D). It allows users to find timeintervals similar to the result (T4). We call this fact-guided
querying, as it enables users to refine extraction results guided bypreviously detected features. To analyze extracted time intervals
in more detail, users can employ the uncertainty-aware TimeTubes
view (T1) as well as multiple linked scatterplots views (E).
Interactive feature analysis interface. Fig. 4 shows our feature and pattern detection user interface. The query specification
panel (A) allows users to build a query with simple interactions
either by selecting what to extract, picking a part of data as
an input, or sketching time variation patterns. After running asimilarity search, TimeTubesX ranks and filters extraction results
according to the parameters in panel (B), and then it displays
all relevant (i.e., non-filtered) extraction results as a collectionof thumbnails in panel (E). The distance distribution histogramin panel (B) helps users to further filter the number of results.
The timeline in panel (C) gives an overview of the temporal
distribution of the results. Users are able to recognize groups ofresults sharing identical data samples and temporal distribution
features. When selecting an individual thumbnail, TimeTubesXshows a detailed information on the corresponding result in panel(D), including exact timestamps and the distance between thequery and the result. To re-utilize the query, compare multiple
query results, and share the query and their results with otherusers, TimeTubesX allows users to export and import queries

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

5

B
C

F

D

A

(a) Light curve plot. The orange diamonds indicate peaks in the data.

E

Fig. 4. The user interface for interactive feature analysis. (A) Query
specification panel; (B) parameters for ranking, filtering, and displaying extraction results and a histogram for the distribution of distances
returned by the similarity search; (C) timelines overviewing displayed
extraction results; (D) a detailed information panel for a selected result;(E) a collection of thumbnails for extraction results; and (F) summary of
the imported previous extraction results.

2

3

1

4
(b) Stokes plane plot. The orange diamonds indicate the peaks in (a).

5
A Anomaly/flare/rotation B Query-by-example

C Query-by-sketch

Fig. 5. Query specification panels for feature and pattern detection
methods. (A) Automatic feature extraction; (B) query-by-example, whereusers can check the selected time interval in a 3D tube view and further
fine-tune query parameters (1) after selecting a time interval in the
TimeTubes or scatterplots views (2); and (C) query-by-sketch, whereusers draw a SOI (3) and assign filtering constraints at each control
point of the hand-drawn sketch (4) or adjust sketch pad settings (5).

and their results in the form of JSON files (a custom format for
TimeTubesX). When importing previous query results, panel (F)shows a summary of the query used in the previous process andpanel (C) shows another timeline for the imported query results.
Fig. 5 shows the query specification panels for each mode of
the feature and pattern detection.
To better demonstrate our feature and pattern detection methods, we will use a synthetic dataset (see Fig. 6) as a running
example in Sections 5 and 6. It contains four large peaks in I, as
highlighted by orange diamonds in (a), three small red circularpatterns on the Stokes plane in (b), three large green rotations ,and three narrow/rough-edged blue patterns.

Fig. 6. Conventional light curve plot and the Stokes plane for our synthetic dataset. The data has three small circular patterns (red), threelarge rotations (green), and three narrow/rough-edged patterns (blue)
within 365 days. Data starts from the square mark and ends at thetriangular mark.

for flares and rotations compared with the ones in our previous
work [7] to identify flares more flexibly and detect rotations
more accurately. Fig. 5 (A) shows the automatic feature extraction
panel, where users can select specific patterns and parameters
for automatic feature extraction. The node (A) in Fig. 3 shows
example TimeTubes views for an anomaly, flare, and rotation.
5.1

Anomaly Detection

It is challenging for astronomers to manually identify time variations across multiple variables. We define data samples withdrastic temporal changes in polarization, intensity, and color as
anomalies. Tracking anomalies can result in identifying unusual
time variations or presages of well-known behaviors, such as flaresor rotations, because observable blazar behaviors show a tendency
to include these drastic variations.
The anomaly degree was defined in our previous work [7] as
the product of the change amount in polarization, intensity, andcolor per day:
Z t+1

5

AUTOMATIC F EATURE E XTRACTION

Astronomers pay careful attention to unusual behaviors of blazars,
such as flare and rotation. As a first step to identify flares androtations (G2), we have implemented automatic feature extraction for the discovery and analysis of dynamic time variations(T3). TimeTubesX automatically extracts three types of features:
anomalies (Section 5.1), flares (Section 5.2), and rotations (Section 5.3). We have significantly improved the detection algorithmstdPolar(t) dI(t) dC(t)
·
·
dt,
dt
dt
dt

where Polar(t) means the position of a data sample on the Stokes
plane at time t, and I(t) and C(t) stand for intensity and color,
respectively.
Experimental results. Applying the anomaly detection to
our synthetic data, data samples around extreme peaks (the orange
diamonds in Fig. 6 (a)) and parts of dynamic rotations (the green
plots in (b)) were highly ranked.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

6

5.3

Fig. 7. Flare detection results for our synthetic data. The results match
the four red data samples in Fig. 6 (a).

(a) Previous rotation detection method [6].

(b) Our improved method.
Fig. 8. Rotation detection results for our synthetic dataset in Fig. 6. The
outline color corresponds to the color in Fig. 6 (b). Our previous method
detected narrow/rough-edged patterns and noise (blue), whereas our
improved method detected only large rotations (green).

5.2

Flare Detection

Flares are defined as extreme peaks of brightness (i.e., emitted
light intensity). Astronomers regard flares as one of the mostimportant observed behaviors of blazars. However, since there
is no specific threshold value of I to define a flare, they needto analyze the local temporal profile of I to identify flares. We
have updated the flare detection methods used in our previous
work [7] to detect relatively small local flares as well as globallylarge flares. To that end, we utilize peak detection methods fortime-series data [40] to extract flare candidates. Flare detection
comprises the following two steps:
Step 1 : Compute the spikiness score S for each data sample;
Step 2 : Filter out data samples with a globally small S.
TimeTubesX uses Equation 1 to compute the spikiness score S for
Step 1 ; we tested multiple equations for S and empirically foundthat the following one produces the best results:∑K (x −x )∑K
k=1 (xi −xi−k )+ k=1 Ki i+k
K

,
(1)
2
where xi denotes a data sample indexed as i and K denotes thenumber of neighbors that should be examined. Equation 1 provides
the average values of the averages of distances between xi and K
left neighbors and those between xi and K right neighbors. Step
2 retains only data samples that satisfy S − S̄ > h × σS , where S̄
and σS denote the mean and standard deviation of all computed
S values for the dataset, respectively, and h is a user-specified
sensitivity threshold. The original algorithm [40] merges peaks
that are close together into one, but we did not adopt this strategy
because astronomers are equally interested in small, individualflares and large, aggregated flares.
Experimental results. We applied the flare detection method
to our synthetic data. We set K and h to 3 and 1, respectively. Fig. 7
presents the flare detection results. The four data samples weredetected as flares, which completely coincide with the deliberately
generated peaks that are highlighted in orange in Fig. 6 (a).
S=

Rotation Detection

Polarization rotation is another important observed behavior of
blazars. Astronomers do not yet agree on whether rotation is anactual feature or just a result of random variations of polarization.
To validate their hypotheses, they scrutinize correlations between
polarization and other properties at the time interval. They typically have been analyzing the time variation of PA to identify
rotations [5], [41], but the rotation center will not be located at theorigin of the Stokes plane ((q, u) = (0, 0)) when there are multiple
polarized components in the sky. Thus, estimating a rotation only
through PA may not allow astronomers to adequately understandits behavior. Our rotation detection is capable of addressing any
rotations regardless of the position of the rotation center.
We use a sliding window approach that allows users tomanually define the length of the time interval for the sliding
window. Based on feedback from astronomers [42], we set thedefault window size to be between three and four weeks.
The computation of the rotation angle is divided into thefollowing seven steps:
Step 1 : Compute the weighted means (q, u) of q and u at the timeinterval ;Step 2 : Compute the standard deviations (σq , σu ) of q and u at thetime interval ;Step 3 : Convert the rectangular coordinates (q − u domain) topolar coordinates (r − θ domain) with its origin shiftedto (q, u);
Step 4 : Filter out time intervals whose σq or σu is smaller than thestandard deviations of the entire dataset;
Step 5 : Compute the difference (θdiff ) of the θ ’s of two consecutive data samples;
Step 6 : Sum θdiff ’s to yield θsum ;Step 7 : Check whether θsum is larger than the user-specified
threshold for the total rotation angle.
We regard q and u as a rotation center. To avoid misleading effectsof outliers and unexpected values at the edges of a time interval,
smaller weights are assigned to both ends of the time interval
according to a Gaussian distribution at Step 1 . Note that users areallowed to adjust these weight ratios. To avoid misclassifying timeintervals with large q or u variance and unlike rotations as rotation
candidates, we have improved upon our previous rotation detection
method [7] on the basis of feedback from two astronomers at
Hiroshima University [43]. To detect only large rotations, our new
method is able to filter out time intervals in which either σq or σu
is smaller than the standard deviation of the entire dataset. Notethat this and other provided filtering constraints also allow for
discovery of small or narrow rotations, such as the red and blue
patterns in Fig. 6 (b). At Step 5 , we cannot compute θdiff simply bysubtracting the θ ’s of consecutive observations due to the range
constraint on θdiff (i.e., θdiff ∈ [0, 2π]). For example, when two
successive data samples are located in the first and fourth quadrantof the Stokes plane, we need to consider whether to take theclockwise or counterclockwise direction as θdiff . We determine
the rotation direction by checking increasing/decreasing tendencyin θ ’s with exponential smoothing [44]. It forecasts the next value
according to past observations by assigning larger weights to more
recent observations. In this way, it addresses the uncertainty about
the rotation direction and make results more feasible (T1). We
estimate the variation trend of θ and then define the angle in
the predicted rotation direction as θdiff . Users can set an arbitrary
angle as a threshold parameter at Step 7 .

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

Experimental results. To compare our novel algorithm withthe previous rotation detection algorithm [7], we applied both toour synthetic data. As the results in Fig. 8 (a) show, the previous
algorithm detected not only large rotations (green patterns in
Fig. 6 (b)) but also narrow or rough-edged patterns (blue). The
improved algorithm more accurately detected only time intervals
in which the polarization values dynamically rotate (green), as
shown in Fig. 8 (b).

6

DYNAMIC V ISUAL Q UERYING

To facilitate astronomers’ discovery of time intervals similar toa ROI/SOI (G3) or those validating specific hypotheses (G4),TimeTubesX provides two different user-initiative visual queryinterfaces: query-by-example (QBE) and query-by-sketch (QBS).
Both are designed to help users identify similar patterns of interest(T4), and our matching algorithm allows for a fuzzy search (T5).
6.1

Query-by-Example

While the TimeTubes view helps users analyze time variations,
it remains challenging for users to build queries for multidimensional, time-dependent data. Our QBE interface allows users
to specify a notable behavior as a ROI with simple interactions
through the TimeTubes or scatterplots views. Users can pick apart of time-series data as an input for a query as well as flexibly
select specific variables that should be queried about to reflect
their intentions. This allows users to easily and intuitively searchlong-term datasets for interesting patterns with minimal required
user inputs.
Fig. 5 (B) shows our QBE interface. To facilitate visual verification by users, a time interval selected through the TimeTubes orscatterplots views in (2, red highlight) is automatically extracted
and displayed in the query panel (1). After selecting the initialtime interval, users can then fine-tune and adjust their query by
selecting the variables that should be queried about. This is crucial
for effective support of queries on multi-dimensional data. Basedon the selected variables, our system interactively updates theappearance of the 3D tube for the selected time interval in (1),visually encoding only currently selected variables. For example,
when users remove the variable C from the query, the tube loses
color variation and becomes a gray tube, as shown in (1). Afterdefining the query, users can adjust the main parameters (e.g.,
normalization and polar coordinates options) for our matchingalgorithm (see Section 6.3).
Experimental results. We verified the effectiveness of our
QBE method using our synthetic data. The red circular patterns in
Fig. 6 (b) have a similar shape but different time lengths, scales,or locations in the Stokes plane. We used the leftmost red timeinterval in (b) as an input query, as shown in Fig. 9 (a). We selected
q and u as active variables and used the normalization and polar
coordinates options to detect time intervals with different scales
or different positions. The length of the target time interval was
set to 5 to 20 days. We used DTWD as a distance function. Notethat we detail the options and parameters related to the matchingprocess in Section 6.3. Fig. 9 (b) shows the top three results of
our QBE in (a). The color of the rectangles corresponds to thoseused in Fig. 6 (b). We were able to detect all time intervals with asimilar shape in our synthetic dataset. Note that the time interval
specified for the query itself is omitted from the results.

7(a) Query.

(b) Results for the query in (a).

Fig. 9. QBE for our synthetic dataset in Fig. 6. We chose the time intervalwith a small circular pattern (the leftmost red pattern in Fig. 6 (b)) for aquery in (a). Our method precisely extracted other time intervals withsmall circular patterns (the two red patterns from the right in Fig. 6 (b)).
The outline color corresponds to the color in Fig. 6 (b).

6.2

Query-by-Sketch

TimeTubesX also allows users to query their data by manuallydrawing time variation patterns onto a 2D sketch interface. Our
QBS interface provides an intuitive and accessible way for users
to specify patterns for multi-dimensional, semi-structured, timedependent blazar datasets and validate their high-level hypotheses.
Rouxel et al. [45] state that an input trajectory by analog gestures
is characterized by three dimensions: space, time, and force. To
realize intuitive interactions, we use space to determine the x andy positions of the stroke and use either drawing speed or drawing
pressure to define the stroke width so that users can describe
time variations of multiple dimensions simultaneously. Using thedrawing speed option, the longer a cursor stays on a single point,
the wider the curve at the point becomes. To take into accountother variables that are not described in sketching gestures, our
QBS interface allows users to assign filtering constraints for eachvariable. Thus, a sketch-based query for multi-dimensional, semistructured, time-dependent data can be built with a single gesture
without sketching time variation patterns several times.
Fig. 5 (C) shows the QBS query specification panel. First,
users define which variables are assigned to the x and y axesof the sketch pad and the stroke width (see panel (5)). Second,
they sketch a time variation pattern on a 2D sketch pad UI (3),where the stroke is shown in black and its width is shown in blue.
Afterward, our system automatically beautifies the input strokeby fitting it to a cubic Bezier curve with as few segments as
possible by using the simplify method in Paper.js [46]. After
their initial drawing, users can further adjust the sketch by adding,removing, or moving control points or by changing the stroke
width. Users are also allowed to assign filtering constraints to eachof the control points on the stroke, as shown in part (4). They candefine value ranges for each variable at the control point, which
the algorithm will subsequently use when evaluating the query.
Experimental results. We evaluated the effectiveness of our
QBS method using our synthetic data. First, we sketched a patternshown in Fig. 10 (a) to extract time intervals with small circularpatterns. The sketch pad plane corresponds to the Stokes plane,
and for simplicity, the stroke width is not used. To detect timeintervals with a similar shape but different positions or scales, weused the normalization and polar coordinates options. Fig. 10 (c)shows the top eight results for our QBS in (a). The outline color ofeach thumbnails matches the color of the corresponding pattern in
Fig. 6 (b). All three time intervals highlighted in red in Fig. 6 (b)were precisely extracted. However, an unexpected result (i.e., thethumbnail with white outline in Fig. 6 (c)) was highly ranked asa candidate similar to the input sketch due to the ambiguities of
the hand-drawn sketch. We address this problem by incorporating
fact-guided querying (see Section 6.4).

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20208

normalize a query and time intervals into the range of [0, 1].
Subsequently, the pattern search places great significance on the
shape of the time variations, whereas the actual value ranges
will be ignored. With the polar coordinates option, q(t) andu(t) are converted into polar coordinates (r − θ domain) before
computing similarities. Subsequently, the pattern search with thenormalization and polar coordinates options will also be able toextract patterns rotating around the origin of the Stoke plane.
(a) A hand-drawn sketch query fora small circular pattern.

(b) A sketch based on the upper
left result in (c).

(c) Results for the query in (a).

(d) Results for the query in (b).

Fig. 10. QBS results for our synthetic dataset in Fig. 6. On the basisof a hand-drawn sketch in (a), an unexpected result is highly ranked(white outline) in (c). With a sketch based on actual data values (b), no
unexpected results appear, as illustrated in (d).

6.3

Query Matching Algorithm

There are many different methods for computing the distance
between two time series, such as Euclidean distance, uniform
scaling, and dynamic time warping (DTW) [47]. TimeTubesX
uses DTW because it aligns two time series elastically and thus
supports the comparison of time series with different lengths. Additionally, it can compute the distance between two time series thatare similar but locally out of phase. According to Eichmann and
Zgraggen [27], the DTW similarity measurement seems to mostclosely match human perception. There are multiple solutions for
dealing with multi-dimensional data in DTW. TimeTubesX implements two options, DTWI (independent) and DTWD (dependent),
both of which are based on the work of Shokoohi-Yekta et al. [48].
DTWI computes the distance between two time series for eachdimension of the data separately. In the final step, it adds up all
the individual distances to produce the final distance measure.
Consequently, this approach focuses more on the similarity of eachdimension and less on correlations between different dimensions.
DTWD , on the other hand, computes the distance between two data
samples directly over all dimensions (i.e., as Euclidean distancein n-dimensional space). Therefore, this method focuses moreon correlations among the different dimensions of a time series
and less on the similarities of individual dimensions. By default,
our system uses DTWD because correlations among variables areimportant for blazar behavior analysis. Readers are recommended
to consult [48, Fig. 1] for comprehensive illustrations of DTWI
and DTWD . We use a sliding window approach in our matchingalgorithm. Users can set the window size and the step size ofthe sliding window and also set a constraint on the largest allowed
temporal shift (i.e., warping window size) in the process of finding
the best alignment between the query and time series. Users can
specify these parameters at the bottom of the query specification
panel in Fig. 4 (A). Note that multiple time intervals with different
lengths but that include the same data samples can be presented
in the extraction results (see Fig. 9 and Fig. 10 (c) and (d)). The
timelines in Fig. 4 (C) help users recognize such time intervals.
Normalization and polar coordinates options enable a fuzzy
pattern search. The normalization option instructs the system to

6.4

Fact-Guided Querying

To support quick and iterative query refinement, TimeTubesX
allows users to re-utilize individual extraction results as an input
for follow-up queries, a process we have termed fact-guided querying. By iteratively updating a query, users can, in a step-by-stepmanner, identify more observable time intervals that reflect theirintentions. Therefore, fact-guided querying contributes immensely
to drilling down into data as a part of the visual exploration
framework of TimeTubesX in Fig. 3. In QBE, users can switch
variables used in the matching process and modify the time rangeof the query, while in QBS, users can adjust the scale, shape,
and axis of the input query or add further filtering constraints.
Thus, fact-guided querying helps users perform further patternsearches based on a time interval of interest found in the previous
process or create a sketch-based query from references to actual
data measurements instead of from a blank sketch pad.
Our system allows users to build a fact-guided query withsimple interactions by, for example, dragging an extraction result
either into the QBE interface (part (2) in Fig. 5 (B)) or into thesketch pad of the QBS interface (part (3) in Fig. 5 (C)).
Experimental results. We confirmed the usefulness ofthe fact-guided querying approach with our synthetic data. As
discussed in Section 6.2, hand-drawn sketch queries, such as
Fig. 10 (a), work well for finding time intervals with a specificfeature, but ambiguities of hand-drawn sketches may sometimes
lead to unexpected results, as shown in (c). If the extraction results
for the query in (a) sufficiently represent users’ intentions, one of
the results can be used as an input to refine the results of the next
visual query. We chose the top left result of Fig. 10 (c) and used
it as an input for our QBS method, as shown in (b). The sketch
pad plane in (b) coincides with the Stokes plane. Thereafter, we
ran QBS using the same parameter settings as the example in
Section 6.2. As shown in the top eight extraction results in (d), thisrefined query in (b) no longer produces any high-ranking outliersor unexpected results.

7

V ISUAL E XPLORATION OF Q UERY R ESULTS

In addition to the feature and pattern detection methods describedin Sections 5 and 6, TimeTubesX includes powerful visual comparison and annotation features for further analysis of blazar data.
7.1

Visual Comparison of Query Results

An essential feature for analysis of blazar datasets is the user’s
ability to compare query results—not just within a single query
but also to previous query results. For example, when users findthat a specific feature frequently appears in a certain time period,
they might want to investigate whether any other features alsofrequently appear in the same time period. Therefore, TimeTubesX
can juxtapose the results of different queries by loading query
results that were previously saved as a JSON file. When importing

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

9

a file, the stored results are mapped to a new timeline thatis arranged as a juxtaposed view below the original timeline.
Hovering over marks on the timeline allows users to see detailed
information about specific results. Additionally, users can re-use orreview the settings of the previous query, such as the selected time
period or the variables assigned to the sketch pad (see Fig. 4 (F)).
7.2

Annotations of Queries and Query Results

To enable efficient collaboration between astronomers and to
facilitate keeping track of analyses between different sessions,
TimeTubesX supports detailed annotations for query results. Users
can access any annotations even after exiting and restarting theapplication because annotations are stored in the local storage of
the web browser. To share annotations with other users, annotations can be exported as a single JSON file. The system stores
the annotation’s timestamp, username, comment, and dataset, as
well as detailed information about the query and query results.
Users can see all of their annotations in a table view or a single
annotation by clicking on a marker with the selected label color
in the TimeTubes and scatterplots views. They can also re-use any
query saved in an annotation by simply clicking on it.
Annotations help users highlight interesting extraction results.
Annotating time intervals of interest not only triggers deeper
inspection of a specific period but also possibly facilitates thediscovery of new features such as periodic patterns.

8(b) A query for an anticorrelated I
and PD variation.

(c) The top twelve time intervals that are similar to the query in (a).

E VALUATIONTimeTubesX is a browser-based application written in HTMLand Javascript. We use React.js to build user interfaces andflux.js to manage the application states. We also utilize standard
libraries such as three.js [49], D3.js [50], and Paper.js [46].
TimeTubesX is open-source (https://github.com/MistletoeNaoko/
TimeTubesWeb), and readers can try the running system using our
synthetic data at https://timetubes.herokuapp.com/.
We shared TimeTubesX with four domain experts, three ofwhom we interviewed for the domain analysis in Section 3.2:
the second author of this paper (Astronomer 1), Astronomer 1’s
former master course student (Astronomer 2), an assistant professor at Hiroshima University (Astronomer 3), and a postdoctoral
researcher at Stanford University (Astronomer 4). Sections 8.1
and 8.2 present two case studies conducted by Astronomer 1, andSection 8.3 discusses the cause of the phenomena presented in
Section 8.2. Section 8.4 reviews qualitative feedback from the fourdomain experts. To examine the magnetic field structures in thejets, astronomers analyze polarimetric observations. Through theanalysis of correlations between polarization and intensity, theyvalidate multiple hypotheses for the increase in the intensity (I).
In the two case studies, Astronomer 1 used the datasets for the
blazar BL Lac. It was empirically noted in [51] that the I of BL
Lac tends to anticorrelate with the polarization degree (PD) duringa certain period. On a MacBook Pro 2017 with a 3.5 GHz Intel
Core i7 and a 16 GB RAM, it took 1,193 ms and 1,236 ms to
obtain the results outlined in Sections 8.1 and 8.2, respectively.
8.1(a) A query for a correlated I and
PD variation.

Case Study 1: Correlation Patterns of I and PD

Astronomer 1’s goal in these case studies was to identify global
statistical features of the period of interest and to build a hypothesis of correlations between the time variations of I and PD.
To achieve this goal, he had to meticulously analyze correlations(d) The top twelve time intervals that are similar to the query in (b).
Fig. 11. Analysis of the blazar BL Lac to investigate correlation patterns
between I and PD. (a) and (b) are user-drawn sketches. (c) and (d) are
the results of QBS with the queries shown in (a) and (b), respectively.

between I and PD in the entire time period and many shorttime intervals within that period. However, it seemed difficult tomanually examine each of the short time intervals. Therefore, hedecided to utilize QBS to investigate correlation patterns in a longterm observation dataset.
First, he formulated a hypothesis that an increase in I tends
to correlate with an increase in PD in BL Lac. To validate thishypothesis and examine how frequently such behavior appears,
he sketched the query shown in Fig. 11 (a), where the x and yaxes express q and u, respectively, and the stroke width represents
I. Therefore, the input sketch indicates a pattern that I gradually
increases and then decreases in a way that is correlated to PD. To
extract time intervals with a similar shape but different position
or different scale, he used the normalization and polar coordinates
options. To avoid missing short events, he set the warping window
size and the sliding window size to 0 and 2, respectively.
Fig. 11 (c) shows the top twelve results of the query sketchedin (a). The timeline at the top of (c) illustrates that the extracted
time intervals seem to be distributed over the entire dataset.
However, in the period from [5,350, 5,450] (enclosed by a redrectangle in (c)), the input pattern seems to occur more frequently

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

than in other periods. Visiting all time intervals in this period
individually and analyzing them in the TimeTubes view, some ofthem include the correlation pattern in (a), but others do not. Thus,
he concluded that the input correlation pattern does not frequently
occur in that period. The reason for this misclassification could bethat our system, when using the normalization option, detects timeintervals with a relatively small variation of I as well. However,
he was still highly pleased with TimeTubesX, as it allowed him toquickly identify a relatively small number of time intervals that hecould then examine in more detail.
8.2

Case Study 2: Anticorrelation Patterns of I and PD

Astronomer 1 also noticed that the variation in I tends to anticorrelate with that in PD. To detect time intervals with such anticorrelated I and PD patterns, he drew another query (Fig. 11 (b)) where
the plane represents the Stokes plane and the stroke width I. The
sketch describes a pattern in which I gradually increases and thendecreases, while PD shows a negative correlation with I. He used
the same parameters as in Section 8.1.
Fig. 11 (d) shows the top twelve results of the query shownin (b). Like in the previous case study, extracted time intervals
seem to be distributed over the entire dataset. However, in the
period from [4,750, 4,850] (enclosed by a red rectangle in (d)),
the input pattern seems to occur more frequently. Astronomer 1’s
group previously reported that, statistically, the variation in I
anticorrelates with that in PD from the years 2008 to 2010 in
BL Lac [51]. However, even in that previous report, they did notanalyze short time intervals in the period individually. By analyzing the extracted time intervals in the period with TimeTubesX,
he could verify not only that an increase in I globally tends toco-occur with a decrease in PD, but also that local peaks of I are
correlated with decreases in PD.
These case studies with QBS underline the importance of local
analysis of short time intervals within a larger time period in
addition to a global analysis of the entire period.
8.3

10

Fig. 12. Comparison of a dataset consisting of q and u (left) and one
consisting of Q and U (right) for the time interval [4,782, 4,890]. In both
images, the position of the data sample goes toward the origin of thedomain (lower left), which means that this polarization variation wascaused by a decrease in the PD of a different polarized component.

component influences the observed Q and U values, meaning thatq and u (fractional values of Q and U) are also influenced. By
comparing the datasets for (q, u) and (Q,U) with the side-byside option in the visual data fusion [6], the plausibility of thesehypotheses can be visually determined.
The following discussion considers the anticorrelation patternsof I and PD, as mentioned in Section 8.2. Astronomer 1 compareda dataset consisting of q and u with one consisting of Q and U.
The comparison results at the time interval with a blue rectangle
displayed in Fig. 11 (d) are shown in Fig. 12. As the red arrows in
Fig. 12 indicate, he found that both data samples move toward theorigin. He found similar behavior at all short time intervals in the
period [4,750, 4,850]. Thus, he finally concluded that the negative
correlation of I and PD in the period is not due to the increase in
unpolarized f lux (Hypothesis 1 ) but is instead due to the presenceof two polarized components (Hypothesis 2 ).
Overall, TimeTubesX greatly facilitated this detailed visual
exploration of blazar data. This allowed Astronomer 1 to examine
many small time intervals with specific features, which would havebeen too laborious with previous methods.

What-If Scenario Analysis

This section explains how astronomers can identify what generally
contributes to the PD variation in blazars in a certain time period.
Astronomers expect that PD variation in a blazar is due to oneof the two following hypotheses:
Hypothesis 1 : total f lux increases (decreases) due to the increase(decrease) in unpolarized f lux;
Hypothesis 2 : There are two polarized components, and they are
perpendicular to each other.
Note that PD can be derived from the amount of f lux:f lux
PD = polarized
total f lux , where total f lux = I = polarized f lux +unpolarized f lux. Astronomers can identify which of the above
hypotheses contributes to the decrease in PD by comparing time
variations of data samples at the time intervals in the q − udomain and in the Q −U domain, where q = Q/I and u = U/I, as
explained in Section 3.1. In the case of Hypothesis 1 , the positionof a data sample in the q − u domain gradually moves toward theorigin ((q, u) = (0, 0)), whereas the position in the Q −U domain
does not move toward the origin ((Q,U) = (0, 0)) because only
the amount of total f lux increases. On the other hand, in the caseof Hypothesis 2 , PD decreases due to a flare of another polarizedcomponent with PA being perpendicular to the jet direction. The
position of a data sample moves toward the origin both in theq − u domain and in the Q −U domain because another polarized

8.4

Qualitative Feedback from Domain Experts

Astronomer 2 found that the rotation detection was useful. With
TimeTubesX’s rotation detection functionality, he was able tofind three unknown rotation behaviors with a shifted rotation
center in the blazar 3C 454.3 [43]. Astronomers 1, 3, and 4
mentioned that the QBS method was especially useful and helpful
for validating their high-level hypotheses, as demonstrated in
Sections 8.1 and 8.2. In particular, Astronomers 3 and 4 said thatdynamic visual querying could be a powerful tool to examine
jet physical processes and behaviors of plasma and magnetic
fields under extreme conditions because the method can efficiently
address arbitrary variation patterns of polarization, intensity, andcolor over time. Astronomers 1 and 3 noticed TimeTubesX’s
potential for data mining. In particular, Astronomer 3 saw massive
potential for mining existing and upcoming large polarization
surveys, while Astronomer 1 found that using TimeTubesX will
enable astronomers to identify interesting features in short timeintervals to an extent that cannot be achieved with conventional
methods due to too many short time intervals in a long-term
dataset. Furthermore, fact-guided querying was impressive to him
because it allows astronomers to refine their sketches based on
actual extracted patterns.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020

9C ONCLUSION AND FUTURE WORK

We have presented a web-based visual analytics environment
for the detailed analysis of blazar datasets, termed TimeTubesX.
TimeTubesX is expected to facilitate astronomers’ analysis ofphotometric and polarimetric behaviors of blazars by enabling
automatic feature extraction and dynamic visual querying. It
allows astronomers not only to easily locate observable blazar
behaviors but also to efficiently find recurring time variation
patterns. TimeTubesX has the potential to allow astronomers tofind short time variation patterns that have not yet been discovered.
In the future, to avoid undesirable effects of outliers, we shouldtake observation errors into account during the feature extraction
processes. We should also consider provenance management toholistically keep track of users’ analysis processes, as realized
in aflak [52]. Furthermore, we would like to incorporate a deeplearning approach to classify time series data in a non-biased
manner because the current query specification process in QBE
and QBS significantly depends on users’ expertise. To provide amore effective overview of the results, it would also be helpful
to cluster the results of a query. Applying TimeTubesX to multidimensional, time-dependent datasets in other domains will likely
form yet another part of our future research.

ACKNOWLEDGMENTS
The authors have benefited from useful discussions with Mahito
Sasada at Hiroshima University, Yannis Liodakis at Stanford
University, and Alan Marscher, Svetlana Jorstad, Manasvita Joshi,
and Zachary Weaver at Boston University. We would like tothank Carolina Nobre at Harvard University for inserting thenarration into the accompanying video. The present work has been
financially supported in part by a MEXT KAKENHI Grant-inAid for Scientific Research(A) No. 17H00737 and King Abdullah
University of Science and Technology (KAUST) and the KAUST
Office of Sponsored Research (OSR)’s award, OSR-2015-CCF2533-0. Sawada, the first author, would also like to thank the
Yoshida Scholarship Foundation.

R EFERENCES
[1]
[2]

[3]
[4]

[5]

[6]

[7]

R. Antonucci, “Unified models for active galactic nuclei and quasars,”Annu. Rev. Astron. Astrophys., vol. 31, pp. 473–521, 1993.
W. Bednarek and R. J. Protheroe, “Gamma-ray and neutrino flares
produced by protons accelerated on an accretion disc surface in active
galactic nuclei,” Mon. Not. R. Astron. Soc., vol. 302, no. 2, pp. 373–380,
1999.
A. Atoyan and C. D. Dermer, “High-energy neutrinos from photomeson
processes in blazars,” Phys. Rev. Lett., vol. 87, no. 22, pp. 221 102:1–
221 102:4, 2001.
A. P. Marscher, S. G. Jorstad, F. D. D’Arcangelo, P. S. Smith, G. G.
Williams, V. M. Larionov, H. Oh, A. R. Olmstead, M. F. Aller, H. D.
Aller, I. M. McHardy, A. Lähteenmäki, M. Tornikoski, E. Valtaoja, V. A.
Hagen-Thorn, E. N. Kopatskaya, W. K. Gear, G. Tosti, O. Kurtanidze,
M. Nikolashvili, L. Sigua, H. R. Miller, and W. T. Ryle, “The inner jetof an active galactic nucleus as revealed by a radio-to-γ-ray outburst,”
Nature, vol. 452, no. 7190, pp. 966–969, 2008.
M. Uemura, R. Itoh, I. Liodakis, D. Blinov, M. Nakayama, L. Xu,
N. Sawada, H.-Y. Wu, and I. Fujishiro, “Optical polarization variationsin the blazar PKS 1749 + 096,” Publ. Astron. Soc. Jpn., vol. 69, no. 6,
pp. 96:1–96:12, 2017.
I. Fujishiro, N. Sawada, M. Nakayama, H.-Y. H.-Y. Wu, K. Watanabe,
S. Takahashi, and M. Uemura, “TimeTubes: Visual exploration of observed blazar datasets,” J. Phys. Conf. Ser., vol. 1036, pp. 012 011:1–
012 011:12, 2018.
N. Sawada, M. Nakayama, M. Uemura, and I. Fujishiro, “TimeTubes:
Automatic extraction of observable blazar features from long-term, multidimensional datasets,” in Proc. IEEE SciVis, 2018, pp. 67–71.

11

[8]

B. R. Kent, “Spherical panoramas for astrophysical data visualization,”
Publ. Astron. Soc. Pac., vol. 129, no. 975, pp. 058 001:1–058 001:10,
2017.
[9] D. Baines, F. Giordano, E. Racero, J. Salgado, B. L. Martı́, B. Merı́n, M.H. Sarmiento, R. Gutiérrez, I. O. de Landaluce, I. León, P. de Teodoro,
J. González, S. Nieto, J. C. Segovia, A. Pollock, M. Rosa, C. Arviset,
D. Lennon, W. O’Mullane, and G. de Marchi, “Visualization of multimission astronomical data with ESASky,” Publ. Astron. Soc. Pac., vol.
129, no. 972, pp. 028 001:1–028 001:6, 2017.
[10] A. Bock, E. Axelsson, J. Costa, G. Payne, M. Acinapura, V. Trakinski,
C. Emmart, C. Silva, C. Hansen, and A. Ynnerman, “OpenSpace: A
System for astrographics,” IEEE Trans. Vis. Comput. Graphics, vol. 26,
no. 1, pp. 633–642, 2020.
[11] K. R. Almryde and A. G. Forbes, “Halos in a dark sky: Interactively
exploring the structure of dark matter halo merger trees,” in Proc. IEEE
SciVis, 2016, pp. 73–77.
[12] H. Li, C.-W. Fu, and A. J. Hanson, “Visualizing multiwavelength astrophysical data,” IEEE Trans. Vis. Comput. Graphics, vol. 14, no. 6, pp.
1555–1562, 2008.
[13] J. N. Burchett, D. Abramov, J. Otto, C. Artanegara, J. X. Prochaska,
and A. G. Forbes, “IGM-Vis: Analyzing intergalactic and circumgalactic
medium absorption using quasar sightlines in a cosmic web context,”
Comput. Graph. Forum, vol. 38, no. 3, pp. 491–504, 2019.
[14] S. Haroz, M. Kwan-Liu, and K. Heitmann, “Multiple uncertainties intime-variant cosmological particle data,” in Proc. IEEE PacificVis, 2008,
pp. 207–214.
[15] A. Preston, R. Ghods, J. Xie, F. Sauer, N. Leaf, K. L. Ma, E. Rangel,
E. Kovacs, K. Heitmann, and S. Habib, “An integrated visualization
system for interactive analysis of large, heterogeneous cosmology data,”
in Proc. IEEE PacificVis, 2016, pp. 48–55.
[16] N. McCurdy and M. Meyer, “GalStamps: Analyzing real and simulated
galaxy observations,” in Proc. IEEE VIS, 2019, pp. 276–280.
[17] H. Hochheiser and B. Shneiderman, “Dynamic query tools for time series
data sets: Timebox widgets for interactive exploration,” Inf. Vis., vol. 3,
no. 1, pp. 1–18, 2004.
[18] P. Buono, A. Aris, C. Plaisant, A. Khella, and B. Shneiderman, “Interactive pattern search in time series,” in Proc. IS&T VDA, vol. 5669, 2005,
pp. 175–186.
[19] P. Buono and A. L. Simeone, “Interactive shape specification for patternsearch in time series,” in Proc. AVI, 2008, pp. 480–481.
[20] C. Holz and S. Feiner, “Relaxed selection techniques for querying timeseries graphs,” in Proc. ACM UIST, 2009, pp. 213–222.
[21] A. Martin and M. Ward, “High dimensional brushing for interactive
exploration of multivariate data,” in Proc. IEEE VIS, 1995, pp. 271–278.
[22] N. Elmqvist, J. Stasko, and P. Tsigas, “DataMeadow: A visual canvas foranalysis of large-scale multivariate data,” in Proc. IEEE VAST, 2007, pp.
187–194.
[23] N. Elmqvist, P. Dragicevic, and J. D. Fekete, “Rolling the Dice: Multidimensional visual exploration using scatterplot matrix navigation,” IEEE
Trans. Vis. Comput. Graphics, vol. 14, no. 6, pp. 1141–1148, 2008.
[24] M. Nielsen, N. Elmqvist, and K. Grønbæk, “Scribble query: Fluid touchbrushing for multivariate data visualization,” in Proc. OzCHI, 2016, pp.
381–390.
[25] M. Wattenberg, “Sketching a graph to query a time-series database,” inProc. ACM CHI EA, 2001, pp. 381–382.
[26] K. Ryall, N. Lesh, T. Lanning, D. Leigh, H. Miyashita, and S. Makino,“Querylines: Approximate query for visual browsing,” in Proc. ACM CHI
EA, 2005, pp. 1765–1768.
[27] P. Eichmann and E. Zgraggen, “Evaluating subjective accuracy in timeseries pattern-matching using human-annotated rankings,” in Proc. ACM
IUI, 2015, pp. 28–37.
[28] P. K. Muthumanickam, K. Vrotsou, M. Cooper, and J. Johansson, “Shapegrammar extraction for efficient query-by-sketch pattern matching in longtime series,” in Proc. IEEE VAST, 2016, pp. 121–130.
[29] N. Ruta, N. Sawada, K. McKeough, M. Behrisch, and J. Beyer, “SAX
Navigator: Time series exploration through hierarchical clustering,” inProc. IEEE VIS, 2019, pp. 236–240.
[30] M. Correll and M. Gleicher, “The semantics of sketch: Flexibility invisual query systems for time series data,” in Proc. IEEE VAST, 2016,
pp. 131–140.
[31] M. Mannino and A. Abouzied, “Expressive time series querying withhand-drawn scale-free sketches,” in Proc. ACM CHI, 2018, pp. 388:1–
388:13.
[32] L. Shao, M. Behrisch, T. Schreck, T. V. Landesberger, M. Scherer,
S. Bremm, and D. Keim, “Guided sketching for visual search andexploration in large scatter plot spaces,” in Proc. EuroVis, 2014, pp. 19–
23.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2020[33] S. Owada, F. Nielsen, and T. Igarashi, “Volume catcher,” in Proc. ACM
I3D, 2005, pp. 111–116.
[34] T. Igarashi, N. Shono, T. Kin, and T. Saito, “Interactive volume segmentation with threshold field painting,” in Proc. ACM UIST, 2016, pp.
403–413.
[35] F.-Y. Tzeng, E. B. Lum, and K.-L. Ma, “A novel interface for higherdimensional classification of volume data,” in Proc. IEEE VIS, 2003, pp.
505–512.
[36] S. Bruckner, V. Šolteszova, E. Groller, J. Hladůvka, K. Buhler, J. Y.
Yu, and B. J. Dickson, “BrainGazer - Visual queries for neurobiology
research,” IEEE Trans. Vis. Comput. Graphics, vol. 15, no. 6, pp. 1497–
1504, 2009.
[37] J. Bernard, J. Brase, D. Fellner, O. Koepler, J. Kohlhammer, T. Ruppert,
T. Schreck, and I. Sens, “A visual digital library approach for timeoriented scientific primary data,” Int. J. Digit. Libr., vol. 11, pp. 111–123,
2010.
[38] D. J.-L. Lee, J. Lee, T. Siddiqui, J. Kim, K. Karahalios, and
A. Parameswaran, “You can’t always sketch what you want: Understanding sensemaking in visual query systems,” IEEE Trans. Vis. Comput.
Graphics, vol. 26, no. 1, pp. 1267–1277, 2020.
[39] M. Uemura, R. Itoh, L. Xu, M. Nakayama, H.-Y. Wu, K. Watanabe,
S. Takahashi, and I. Fujishiro, “TimeTubes: Visualization of polarization
variations in blazars,” Galaxies, vol. 4, no. 3, pp. 23:1–23:9, 2016.
[40] G. K. Palshikar, “Simple algorithms for peak detection in time-series,” inProc. IIMA ICADABAI, 2009, pp. 1–13.
[41] Y. Ikejiri, M. Uemura, M. Sasada, R. Ito, M. Yamanaka, K. Sakimoto,A. Arai, Y. Fukazawa, T. Ohsugi, K. S. Kawabata, M. Yoshida, S. Sato,and M. Kino, “Photopolarimetric monitoring of blazars in the optical andnear-infrared bands with the Kanata telescope. I. Correlations between
flux, color, and polarization,” Publ. Astron. Soc. Jpn., vol. 63, no. 3, pp.
639–675, 2011.
[42] M. Sasada, M. Uemura, Y. Fukazawa, K. S. Kawabata, R. Itoh, I. Sakon,
K. Fujisawa, A. Kadota, T. Ohsugi, M. Yoshida, H. Yasuda, M. Yamanaka, S. Sato, and M. Kino, “Multi-wavelength photometric andpolarimetric observations of the outburst of 3C 454.3 in 2009 December,”Publ. Astron. Soc. Jpn., vol. 64, no. 3, pp. 58:1–58:8, 2012.
[43] H. Ruochen, Study of the Magnetic Field in the Jet of Active
Galactic Nuclei Using a New Visualization Technique of Polarization
Data. Hiroshima University, 2019, [in Japanese]. [Online]. Available:
http://www-heaf.astro.hiroshima-u.ac.jp/thesis/huang2018.pdf[44] R. G. Brown, Exponential smoothing for predicting demand. Little,
1956.
[45] B. Rouxel, F. Poirier, J. Y. Antoine, and G. Coppin, “What you draw iswhat you search: The analog gesture,” in Proc. HCII, 2014, pp. 139–147.
[46] J. Lehni and J. Puckey, “Paper.js,” 2020. [Online]. Available:
http://paperjs.org/[47] D. J. Bemdt, D. J. Berndt, and J. Clifford, “Using dynamic time warpingto find patterns in time series,” in Proc. AAAIWS, vol. 398, 1994, pp.
359–370.
[48] M. Shokoohi-Yekta, J. Wang, and E. Keogh, “On the non-trivial generalization of Dynamic Time Warping to the multi-dimensional case,” inProc. SDM, 2015, pp. 289–297.
[49] three.js, “three.js,” 2020. [Online]. Available: https://threejs.org/[50] M. Bostock, “D3.js,” 2020. [Online]. Available: https://d3js.org/[51] H. Gaur, A. C. Gupta, P. J. Wiita, M. Uemura, R. Itoh, and M. Sasada,“Anti-correlated optical flux and polarization variability in BL Lac,”
Astrophys. J. Lett., vol. 781, no. 1, pp. L4:1–L4:5, 2014.
[52] M. O. Boussejra, R. Uchiki, Y. Takeshima, K. Matsubayashi,
S. Takekawa, M. Uemura, and I. Fujishiro, “aflak: Visual programming
environment enabling end-to-end provenance management for the analysis of astronomical datasets,” Vis. Inform., vol. 3, no. 1, pp. 1–8, 2019.

Naoko Sawada received BE and ME degrees
in information and computer science from Keio
University in 2017 and 2018, respectively. She
is currently working toward a PhD degree at
the School of Science for Open and Environmental Systems, Keio University. She stayed at
Harvard University as a visiting scholar in the
Visual Computing Group from 2018 to 2020. Her
main research interests are time-varying multidimensional data visualization and analytics.

12

Makoto Uemura received a PhD degree from
Kyoto University, Japan, in 2004 for researchon astrophysics. He is an associate professor
at Hiroshima Astrophysical Science Center, Hiroshima University. His research interests include physics of astronomical variable stars,
data science, and visual analytics for astronomy. He is especially interested in the analysis
of high-dimensional data concerning black holesystems.

Johanna Beyer is a research associate in the
School of Engineering and Applied Sciences at
Harvard University. Before joining Harvard, she
was a postdoctoral fellow at the Geometric Modeling and Scientific Visualization Center at King
Abdullah University of Science and Technology.
Her research interests include large-data visualization, GPU-based volume rendering, and the
combination of abstract information visualization with scientific visualization for novel domainspecific applications. She received a PhD incomputer science from the Vienna University of Technology in 2010.

Hanspeter Pfister is the An Wang Professor ofComputer Science at the Harvard John A. Paulson School of Engineering and Applied Sciences
and an affiliate faculty member of the Center for
Brain Science. His research in visual computing
lies at the intersection of visualization, computer
graphics, and computer vision and spans a wide
range of topics, including biomedical image analysis and visualization, image and video analysis, interpretable machine learning, and visual
analytics in data science. Pfister has a PhD incomputer science from the State University of New York at Stony Brook
and an MS in electrical engineering from ETH Zurich, Switzerland.
From 2013 to 2017, he was the director of the Institute for Applied
Computational Science. Before joining Harvard, he worked for over a
decade at Mitsubishi Electric Research Laboratories, where he wasan associate director and senior research scientist. He was the chiefarchitect of VolumePro, Mitsubishi Electric’s award-winning real-time
volume rendering graphics card, for which he received the Mitsubishi
Electric President’s Award in 2000. Pfister was elected as an ACM
Fellow in 2019.

Issei Fujishiro is a professor at the Department
of Information and Computer Science, Faculty of
Science and Technology, Keio University. He is
also an adjunct professor at the School of Computer Science and Technology, Hangzhou Dianzi University. Before joining Keio University in
2009, he held faculty positions at the Universityof Tokyo, University of Tsukuba, Ochanomizu
University, and Tohoku University. He stayed as
a visiting professor at the State University of
New York at Stony Brook from 1994 to 1995.
He received a Doctor of Science degree in information sciences fromthe University of Tokyo in 1988. His research interests include modeling paradigms and shape representations, applied visualization designand lifecycle management, and smart ambient media with multi-modal
displays. He has served as an associate editor for IEEE TVCG and the
Elsevier Journal of Visual Informatics. He has chaired 35 international
conferences on visualization and computer graphics, acting as a paper
co-chair (SciVis) for IEEE VIS 2019/2018 and as a workshop co-chair
for PacificVAST 2018 and TopoInVis 2017. Fujishiro is a member of the
Science Council of Japan.