prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What factors control the reliance of artificial organisms on plasticity?

Read the following text and answer briefly.

Context 1: Paper Info

Title: Environmental variability and network structure determine the optimal plasticity mechanisms in embodied agents
Publish Date: Unkown
Author List: Sina Khajehabdollahi (from Department of Computer Science, University of Tübingen)

Figure

Figure2: An outline of the network controlling the foraging agent The sensor layer receives inputs at each time step (the ingredients of the nearest food), which are processed by the plastic layer in the same way as the static sensory network, Fig 1 The output of that network is given as input to the motor network, along with the distance d and angle α to the nearest food, the current velocity v, and energy E of the agent These signals are processed through two hidden layers to the final output of motor commands as the linear and angular acceleration of the agent
Figure4: The evolved parameters θ = (θ 1 ,      , θ 8 ) of the plasticity rule for the reward prediction (a ) and the decision (b ) tasks, for a variety of parameters (p tr = 0 01, d e ∈ 0, 0 1,      , 1, and σ ∈ 0, 0 1,      , 1 in all 100 combinations) Despite the relatively small difference between the tasks, the evolved learning rules differ considerably For visual guidance, the lines connect θs from the same run 
Figure5: a The trajectory of an agent (blue line) in the 2D environment A well-trained agent will approach and consume food with positive values (green dots) and avoid negative food (red dots) b The learning rate of the plastic sensory network eta p grows with the distance between environments d e c  and decreases with the frequency of environmental change d The fitness of an agent (measured as the total food consumed over its lifetime) increases over generations of the EA for both the scalar and binary readouts in the sensory network e The Pearson correlation coefficient of an evolved agent's weights with the ingredient value vector of the current environment (E 1 -blue, E 2 -red) In this example, the agent's weights are anti-correlated with its environment, which is not an issue for performance since the motor network can interpret the inverted signs of food 

abstract

The evolutionary balance between innate and learned behaviors is highly intricate, and different organisms have found different solutions to this problem  We hypothesize that the emergence and exact form of learning behaviors is naturally connected with the statistics of environmental fluctuations and tasks an organism needs to solve 
Here, we study how different aspects of simulated environments shape an evolved synaptic plasticity rule in static and moving artificial agents  We demonstrate that environmental fluctuation and uncertainty control the reliance of artificial organisms on plasticity  Interestingly, the form of the emerging plasticity rule is additionally determined by the details of the task the artificial organisms are aiming to solve 
Moreover, we show that coevolution between static connectivity and interacting plasticity mechanisms in distinct sub-networks changes the function and form of the emerging plasticity rules in embodied agents performing a foraging task  One of the defining features of living organisms is their ability to adapt to their environment and incorporate new information to modify their behavior 
It is unclear how the ability to learn first evolved , but its utility appears evident  Natural environments are too complex for all the necessary information to be hardcoded genetically and more importantly, they keep changing during an organism's lifetime in ways that cannot be anticipated ;   The link between learning and environmental uncertainty and fluctuation has been extensively demonstrated in both natural ; , and artificial environments  
Nevertheless, the ability to learn does not come without costs  For the capacity to learn to be beneficial in evolutionary terms, a costly nurturing period is often required, a phenomenon observed in both biological , and artificial organisms   Additionally, it has been shown that in some complex environments, hardcoded behaviors may be superior to learned ones given limits in the agent's lifetime and envi-ronmental uncertainty ; ;  
The theoretical investigation of the optimal balance between learned and innate behaviors in natural and artificial systems goes back several decades  However, it has recently found also a wide range of applications in applied AI systems ;   Most AI systems are trained for specific tasks, and have no need for modification after their training has been completed 
Still, technological advances and the necessity to solve broad families of tasks make discussions about life-like AI systems relevant to a wide range of potential application areas  Thus the idea of open-ended AI agents that can continually interact with and adapt to changing environments has become particularly appealing 
Many different approaches for introducing lifelong learning in artificial agents have been proposed  Some of them draw direct inspiration from actual biological systems ;   Among them, the most biologically plausible solution is to equip artificial neural networks with some local neural plasticity , similar to the large variety of synaptic plasticity mechanisms ; ; that performs the bulk of the learning in the brains of living organisms  
The artificial plasticity mechanisms can be optimized to modify the connectivity of the artificial neural networks toward solving a particular task  The optimization can use a variety of approaches, most commonly evolutionary computation  The idea of meta-learning or optimizing synaptic plasticity rules to perform specific functions has been recently established as an engineering tool that can compete with stateof-the-art machine learning algorithms on various complex tasks ; ; Pedersen and Risi (2021);  
Additionally, it can be used to reverse engineer actual plasticity mechanisms found in biological neural networks and uncover their functions ;   Here, we study the effect that different factors (environ-arXiv:2303 06734v1 [q-bio NC] 12 Mar 2023 mental fluctuation and reliability, task complexity) have on the form of evolved functional reward-modulated plasticity rules 
We investigate the evolution of plasticity rules in static, single-layer simple networks  Then we increase the complexity by switching to moving agents performing a complex foraging task  In both cases, we study the impact of different environmental parameters on the form of the evolved plasticity mechanisms and the interaction of learned and static network connectivity 
Interestingly, we find that different environmental conditions and different combinations of static and plastic connectivity have a very large impact on the resulting plasticity rules  We imagine an agent who must forage to survive in an environment presenting various types of complex food particles  Each food particle is composed of various amounts and combinations of N ingredients that can have positive (food) or negative (poison) values 
The value of a food particle is a weighted sum of its ingredients  To predict the reward value of a given resource, the agent must learn the values of these ingredients by interacting with the environment  The priors could be generated by genetic memory, but the exact values are subject to change  To introduce environmental variability, we stochastically change the values of the ingredients 
More precisely, we define two ingredient-value distributions E 1 and E 2 and switch between them, with probability p tr for every time step  We control how (dis)similar the environments are by parametrically setting E 2 = (1 − 2d e )E 1 , with d e ∈ [0, 1] serving as a distance proxy for the environments; when d e = 0, the environment remains unchanged, and when d e = 1 the value of each ingredient fully reverses when the environmental transition happens 
For simplicity, we take values of the ingredients in E 1 equally spaced between -1 and 1 (for the visualization, see Fig  )  The static agent receives passively presented food as a vector of ingredients and can assess its compound value using the linear summation of its sensors with the (learned or evolved) weights, see Fig   
The network consists of N sensory neurons that are projecting to a single post-synaptic neuron  At each time step, an input X t = (x 1 ,       , x N ) is presented, were the value x i , i ∈ {1,       , N } represents the quantity of the ingredient i  We draw x i independently form a uniform distribution on the [0, 1] interval (x i ∼ U (0, 1)) 
The value of each ingredient w c i is determined by the environment (E 1 or E 2 )  The postsynaptic neuron outputs a prediction of the food X t value as y t = g(W X T t )  Throughout the paper, g will be either the identity function, in which case the prediction neuron is linear, or a step-function; however, it could be any other nonlinearity, such as a sigmoid or ReLU 
After outputting the prediction, the neuron receives feedback in the form of the real value of the input R t   The real value is computed as R t = W c X T t + ξ, where W c = (w c 1 ,       , w c N ) is the actual value of the ingredients, and ξ is a term summarizing the noise of reward and sensing system ξ ∼ N (0, σ) 
Figure : An outline of the static agent's network  The sensor layer receives inputs representing the quantity of each ingredient of a given food at each time step  The agent computes the prediction of the food's value y t and is then given the true value R t ; it finally uses this information in the plasticity rule to update the weight matrix 
For the evolutionary adjustment of the agent's parameters, the loss of the static agent is the sum of the mean squared errors (MSE) between its prediction y t and the reward R t over the lifetime of the agent  The agent's initial weights are set to the average of the two ingredient value distributions, which is the optimal initial value for the case of symmetric switching of environments that we consider here 
As a next step, we incorporate the sensory network of static agents into embodied agents that can move around in an environment scattered with food  To this end, we merge the static agent's network with a second, non-plastic motor network that is responsible for controlling the motion of the agent in the environment 
Specifically, the original plastic network now provides the agent with information about the value of the nearest food  The embodied agent has additional sensors for the distance from the nearest food, the angle between the current velocity and the nearest food direction, its own velocity, and its own energy level (sum of consumed food values) 
These inputs are processed by two hidden layers (of 30 and 15 neurons) with tanh activation  The network's outputs are angular and linear acceleration, Fig    The embodied agents spawn in a 2D space with periodic boundary conditions along with a number of food particles that are selected such that the mean of the food value distribution is ∼ 0  An agent can eat food by approaching it sufficiently closely, and each time a food particle is eaten, it is The sensor layer receives inputs at each time step (the ingredients of the nearest food), which are processed by the plastic layer in the same way as the static sensory network, Fig   
The output of that network is given as input to the motor network, along with the distance d and angle α to the nearest food, the current velocity v, and energy E of the agent  These signals are processed through two hidden layers to the final output of motor commands as the linear and angular acceleration of the agent re-spawned with the same value somewhere randomly on the grid (following the setup of ) 
After 5000 time steps, the cumulative reward of the agent (the sum of the values of all the food it consumed) is taken as its fitness  During the evolutionary optimization, the parameters for both the motor network (connections) and plastic network (learning rule parameters) are co-evolved, and so agents must simultaneously learn to move and discriminate good/bad food 
Reward-modulated plasticity is one of the most promising explanations for biological credit assignment   In our network, the plasticity rule that updates the weights of the linear sensor network is a rewardmodulated rule which is parameterized as a linear combination of the input, the output, and the reward at each time step:
Additionally, after each plasticity step, the weights are normalized by mean subtraction, an important step for the stabilization of Hebbian-like plasticity rules   We use a genetic algorithm to optimize the learning rate η p and amplitudes of different terms θ = (θ 1 ,       , θ 8 )  The successful plasticity rule after many food presentations must converge to a weight vector that predicts the correct food values (or allows the agent to correctly decide whether to eat a food or avoid it) 
To have comparable results, we divide θ = (θ 1 ,       , θ 8 ) by We then multiply the learning rate η p with θ max to maintain the rule's evolved form unchanged, η norm p = η p • θ max   In the following, we always use normalized η p and θ, omitting norm   To evolve the plasticity rule and the moving agents' motor networks, we use a simple genetic algorithm with elitism  
The agents' parameters are initialized at random (drawn from a Gaussian distribution), then the sensory network is trained by the plasticity rule and finally, the agents are evaluated  After each generation, the bestperforming agents (top 10 % of the population size) are selected and copied into the next generation 
The remaining 90 % of the generation is repopulated with mutated copies of the best-performing agents  We mutate agents by adding independent Gaussian noise (σ = 0 1) to its parameters  To start with, we consider a static agent whose goal is to identify the value of presented food correctly  The static reward-prediction network quickly evolves the parameters of the learning rule, successfully solving the prediction task 
We first look at the evolved learning rate η p , which determines how fast (if at all) the network's weight vector is updated during the lifetime of the agents  We identify three factors that control the learning rate parameter the EA converges to: the distance between the environments, the noisiness of the reward, and the rate of environmental transition 
The first natural factor is the distance d e between the two environments, with a larger distance requiring a higher learning rate, Fig    This is an expected result since the convergence time to the "correct" weights is highly dependent on the initial conditions  If an agent is born at a point very close to optimality, which naturally happens if the environments are similar, the distance it needs to traverse on the fitness landscape is small 
Therefore it can afford to have a small learning rate, which leads to a more stable convergence and is not affected by noise  A second parameter that impacts the learning rate is the variance of the rewards  The reward an agent receives for the plasticity step contains a noise term ξ that is drawn from a zero mean Gaussian distribution with standard deviation σ 
This parameter controls the unreliability of the agent's sensory system, i e , higher σ means that the information the agent gets about the value of the foods it consumes cannot be fully trusted to reflect the actual value of the foods  As σ increases, the learning rate η p decreases, which means that the more unreliable an environment becomes, the less an agent relies on plasticity to update its weights, Fig   
Indeed for some combinations of relatively small distance d e and high reward variance σ, the EA converges to a learning rate of η p ≈ 0  This means that the agent opts to have no adaptation during its lifetime and remain at the mean of the two environments  It is an optimal solution when the expected loss due to ignoring the environmental transitions is, on average, lower than the loss the plastic network will incur by learning via the (often misleading because of the high σ) environmental cues 
A final factor that affects the learning rate the EA will converge to is the frequency of environmental change during an agent's lifetime  Since the environmental change is modeled as a simple, two-state Markov process (Fig  ), the control parameter is the transition probability p tr   When keeping everything else the same, the learning rate rapidly rises as we increase the transition probability from 0, and after reaching a peak, it begins to decline slowly, eventually reaching zero (Fig  ) 
This means that when environmental transition is very rare, agents opt for a very low learning rate, allowing a slow and stable convergence to an environment-appropriate weight vector that leads to very low losses while the agent remains in that environment  As the rate of environmental transition increases, faster learning is required to speed up convergence in order to exploit the (comparatively shorter) stays in each environment 
Finally, as the environmental transition becomes too fast, the agents opt for slower or even no learning, which keeps them ) and the decision (b ) tasks, for a variety of parameters (p tr = 0 01, d e ∈ 0, 0 1,       , 1, and σ ∈ 0, 0 1,       , 1 in all 100 combinations)  Despite the relatively small difference between the tasks, the evolved learning rules differ considerably 
For visual guidance, the lines connect θs from the same run  near the middle of the two environments, ensuring that the average loss of the two environments is minimal (Fig  )  The form of the evolved learning rule depends on the task: Decision vs  Prediction The plasticity parameters θ = (θ 1 ,       , θ 8 ) for the rewardprediction task converge on approximately the same point, regardless of the environmental parameters (Fig  ) 
In particular, θ 3 → 1, θ 5 → −1, θ i → 0 for all other i, and thus the learning rule converges to: Since by definition y t = g(W t X T t ) = W t X T t (g(x) = x in this experiment) and R t = W c X T t + ξ we get: Thus the distribution of ∆W t converges to a distribution with mean 0 and variance depending on η p and σ and W converges to W c  
So this learning rule will match the agent's weight vector with the vector of ingredient values in the environment  We examine the robustness of the learning rule the EA discovers by considering a slight modification of our task  Instead of predicting the expected food value, the agent now needs to decide whether to eat the presented food or not 
This is done by introducing a step-function nonlinearity (g(x) = 1 if x ≥ 1 and 0 otherwise)  Then the output y(t) is computed as: Instead of the MSE loss between prediction and actual value, the fitness of the agent is now defined as the sum of the food values it chose to consume (by giving y t = 1)  Besides these two changes, the setup of the experiments remains exactly the same 
The qualitative relation between η p and parameters of environment d e , σ and p tr is preserved in the changed experiment  However, the resulting learning rule is significantly different (Fig  )  The evolution converges to the following learning rule: In both cases, the rule has the form ∆W t = η p X t [α y R t + β y ] 
Thus, the ∆W t is positive or negative depending on whether the reward R t is above or below a threshold (γ = −β y /α y ) that depends on the output decision of the network (y t = 0 or 1)  Both learning rules (for the reward-prediction and decision tasks) have a clear Hebbian form (coordination of preand post-synaptic activity) and use the incoming reward signal as a threshold 
These similarities indicate some common organizing principles of reward-modulated learning rules, but their significant differences highlight the sensitivity of the optimization process to task details  We now turn to the moving embodied agents in the 2D environment  To optimize these agents, both the motor network's connections and the sensory network's plasticity parameters evolve simultaneously 
Since the motor network is initially random and the agent has to move to find food, the number of interactions an agent experiences in its lifetime can be small, slowing down the learning  However, having the larger motor network also has benefits for evolution because it allows the output of the plastic network to be read out and transformed in different ways, resulting in a broad set of solutions 
The fitness of an agent (measured as the total food consumed over its lifetime) increases over generations of the EA for both the scalar and binary readouts in the sensory network  e  The Pearson correlation coefficient of an evolved agent's weights with the ingredient value vector of the current environment (E 1 -blue, E 2 -red) 
In this example, the agent's weights are anti-correlated with its environment, which is not an issue for performance since the motor network can interpret the inverted signs of food  The agents can solve the task effectively by evolving a functional motor network and a plasticity rule that converges to interpretable weights (Fig  ) 
After ∼ 100 evolutionary steps (Fig  ), the agents can learn the ingredient value distribution using the plastic network and reliably move towards foods with positive values while avoiding the ones with negative values  We compare the dependence of the moving and the static agents on the parameters of the environment: d e and the state transition probability p tr  
At first, in order to simplify the experiment, we set the transition probability to 0, but fixed the initial weights to be the average of E 1 and E 2 , while the real state is E 2   In this experiment, the distance between states d e indicates twice the distance between the agent's initial weights and the optimal weights (the environment's ingredient values) since the agent is initialized at the mean of the two environment distributions 
Same as for the static agent, the learning rate increases with the distance d e (Fig  )  Then, we examine the effect of the environmental transition probability p tr on the evolved learning rate η p   In order for an agent to get sufficient exposure to each environment, we scale down the probability p tr from the equivalent experiment for the static agents 
We find that as the probability of transition increases, the evolved learning rate η p decreases (Fig  )  This fits with the larger trend for the static agent, although there is a clear difference when it comes to the increase for very small transition probabil-ities that were clearly identifiable in the static but not the moving agents 
This could be due to much sparser data and possibly the insufficiently long lifetime of the moving agent (the necessity of scaling makes direct comparisons difficult)  Nevertheless, overall we see that the associations observed in the static agents between environmental distance d e and transition probability p tr and the evolved learning rate η p are largely maintained in the moving agents 
Still, more data would be needed to make any conclusive assertions about the exact effect of these environmental parameters on the emerging plasticity mechanisms  A crucial difference between the static and the moving agents is the function the plasticity has to perform  While in the static agents, the plasticity has to effectively identify the exact value distribution of the environment in order to produce accurate predictions, in the embodied agents, the plasticity has to merely produce a representation of the environment that the motor network can evolve to interpret adequately enough to make decisions about which food to consume 
To illustrate the difference, we plot the Pearson correlation coefficient between an agent's weights and the ingredient values of the environment it is moving in (Fig  )  We use the correlation instead of the MSE loss (which we used for the static agents in Fig  ) because the amplitude of the vector varies a lot for different agents and meaningful The evolved parameters of moving agents' plasticity rule for the g(s) = x, identity (a ) and the step function (Eq 
4) (b ) sensory networks (the environmental parameters here are d e ∈ [0, 1], σ = 0 and p tr = 0 001)  The step function (binary output) network evolved a more structured plasticity rule (e g , θ 3 > 0 for all realizations) than the linear network  Moreover, the learned weights for the identity network (c ) have higher variance and correlate significantly less with the environment's ingredient distribution compared to the learned weights for the thresholded network (d )
conclusions cannot be drawn from the MSE loss  For many agents, the learned weights are consistently anti-correlated with the actual ingredient values (an example of such an agent is shown in Fig  )  This means that the output of the sensory network will have the opposite sign from the actual food value 
While in the static network, this would lead to very bad predictions and high loss, in the foraging task, these agents perform exactly as well as the ones where the weights and ingredients values are positively correlated, since the motor network can simply learn to move towards food for which it gets a negative instead of a positive sensory input 
This additional step of the output of the plastic network going through the motor network before producing any behavior has a strong effect on the plasticity rules that the embodied agents evolve  Specifically, if we look at the emerging rules the top performing agents have evolved (Fig  ), it becomes clear that, unlike the very well-structured rules of the static agents (Fig  ), there is now virtually no discernible pattern or structure 
The difference becomes even clearer if we look at the learned weights (at the end of a simulation) of the best-performing agents (Fig  )  While there is some correlation with the environment's ingredient value distribution, the variance is very large, and they do not seem to converge on the "correct" values in any way 
This is to some extent expected since, unlike the static agents where the network's output has to be exactly correct, driving the evolution of rules that converge to the precise environmental distribution, in the embodied networks, the bulk of the processing is done by the motor network which can evolve to interpret the scalar value of the sensory network's output in a variety of ways 
Thus, as long as the sensory network's plasticity rule co-evolves with the motor network, any plasticity rule that learns to produce consistent information about the value of encountered food can potentially be selected  To further test this assumption, we introduce a bottleneck of information propagation between the sensory and motor networks by using a step-function nonlinearity on the output of the sensory network (Eq 
4)  Similarly to the decision task of the static network, the output of the sensory network now becomes binary  This effectively reduces the flow of information from the sensory to the motor network, forcing the sensory network to consistently decide whether food should be consumed (with the caveat that the motor network can still interpret the binary sign in either of two ways, either consuming food marked with 1 or the ones marked with 0 by the sensory network) 
The agents perform equally well in this variation of the task as before (Fig  ), but now, the evolved plasticity rules seem to be more structured (Fig  )  Moreover, the variance of the learned weights in the bestperforming agents is significantly reduced (Fig  ), which indicates that the bottleneck in the sensory network is in-creasing selection pressure for rules that learn the environment's food distribution accurately 
We find that different sources of variability have a strong impact on the extent to which evolving agents will develop neuronal plasticity mechanisms for adapting to their environment  A diverse environment, a reliable sensory system, and a rate of environmental change that is neither too large nor too small are necessary conditions for an agent to be able to effectively adapt via synaptic plasticity 
Additionally, we find that minor variations of the task an agent has to solve or the parametrization of the network can give rise to significantly different plasticity rules  Our results partially extend to embodied artificial agents performing a foraging task  We show that environmental variability also pushes the development of plasticity in such agents 
Still, in contrast to the static agents, we find that the interaction of a static motor network with a plastic sensory network gives rise to a much greater variety of wellfunctioning learning rules  We propose a potential cause of this degeneracy; as the relatively complex motor network is allowed to read out and process the outputs from the plastic network, any consistent information coming out of these outputs can be potentially interpreted in a behaviorally useful way 
Reducing the information the motor network can extract from the sensory system significantly limits learning rule variability  Our findings on the effect of environmental variability concur with the findings of previous studies that have identified the constraints that environmental variability places on the evolutionary viability of learning behaviors 
We extend these findings in a mechanistic model which uses a biologically plausible learning mechanism (synaptic plasticity)  We show how a simple evolutionary algorithm can optimize the different parameters of a simple reward-modulated plasticity rule for solving simple prediction and decision tasks 
Reward-modulated plasticity has been extensively studied as a plausible mechanism for credit assignment in the brain ; ; and has found several applications in artificial intelligence and robotics tasks ;   Here, we demonstrate how such rules can be very well-tuned to take into account different environmental parameters and produce optimal behavior in simple systems 
Additionally, we demonstrate how the co-evolution of plasticity and static functional connectivity in different subnetworks fundamentally changes the evolutionary pressures on the resulting plasticity rules, allowing for greater diversity in the form of the learning rule and the resulting learned connectivity 
Several studies have demonstrated how, in biological networks, synaptic plasticity heavily interacts with and is driven by network topology   Moreover, it has been recently demonstrated that biological plasticity mechanisms are highly redundant in the sense that any observed neural connectivity or recorded activity can be achieved with a variety of distinct, unrelated learning rules  
This observed redundancy of learning rules in biological settings complements our results and suggests that the function of plasticity rules cannot be studied independently of the connectivity and topology of the networks they are acting on  The optimization of functional plasticity in neural networks is a promising research direction both as a means to understand biological learning processes and as a tool for building more autonomous artificial systems 
Our results suggest that reward-modulated plasticity is highly adaptable to different environments and can be incorporated into larger systems that solve complex tasks  This work studies a simplified toy model of neural network learning in stochastic environments  Future work could be built on this basic framework to examine more complex reward distributions and sources of environmental variability 
Moreover, a greater degree of biological realism could be added by studying more plausible network architectures (multiple plastic layers, recurrent and feedback connections) and more sophisticated plasticity rule parametrizations  Additionally, our foraging simulations were constrained by limited computational resources and were far from exhaustive 
Further experiments can investigate environments with different constraints, food distributions, multiple seasons, more complex motor control systems and interactions of those systems with different sensory networks as well as the inclusion of plasticity on the motor parts of the artificial organisms 
Context 2: Paper Info

Title: Incorporating Human Path Preferences in Robot Navigation with Minimal Interventions
Publish Date: 16 Mar 2023
Author List: Oriana Peltzer, Dylan Asmar, Mac Schwager, Mykel Kochenderfer

Figure

Hyperplane arrangement of a twodimensional space containing two obstacles (colored in gray) The robot is located inside the pink polytope, surrounded by three adjacent obstacle-free polytopes Each hyperplane on the boundary of the robot's polytope corresponds to one of the nonredundant constraints in eq (4) (b)Graph derived from the hyperplane arrangement The nodes on the graph designate polytopes, and edges designate transitions to adjacent polytopes To estimate the human's preference, the robot updates a posterior over the goal and over which of the graph transitions φ 1 , φ 2 and φ 3 is preferred by the human (c)Example preference defined over the graph The location of the goal is indicated in yellow in the lower right polytope For each node, the outgoing pink arrow designates the edge on the graph corresponding to the preferred transition between polytopes 
Simple, 10 × 10, 8 polytopes (b) Map 2: Office, 10 × 10, 56 polytopes (c) Map 3: Classroom, 20 × 20, 73 polytopes (d) Sampled observations and robot's executed trajectories 
Fig 5: Maps used for simulating the robot navigation problem with path preferences In (d), the heading angles observed are indicated with arrows The goal is indicated with a pink circle, and the orange robot corresponds to the starting location The blue robot follows a policy that accounts for path preference, while the green robot does not The opacity of the robots increases with time 
Map 1 problem setup and example realizations for goal-only (green) and path preference (blue) solution methods The robot starts at the lower left corner of the environment, and the goal of the task (pink circle) is in the upper left area The robot does not know which goal, among 10 options (shown in light blue squares), is the correct goal The human provides noisy observations, indicated by arrows, at each iteration The green robot selects actions according to the goal-only baseline, and the blue robot uses our proposed method to infer path preferences The polytopes composing G are drawn in blue Probability of correct goal WLPHVWHS +J (c) Entropy of goal distribution g 
Fig  6: Probability of the correct goal, fig 6b, and entropy of the goal belief distribution P (g), fig 6c, for the same problem setup, fig 6a In this problem instance, the human's preference is to go to the goal by passing on the right side of the obstacle Results are averaged over 50 runs and the area filled represents one standard deviation above and below the mean value The goal-only baseline shows an over-confident prediction (shown by the strong reduction in belief entropy) that the correct goal is less likely, making it more difficult to reach the correct goal compared to a method that accounts for path preference 
Success rates in the simple environment (Map 1) The results are averaged over 6 randomly sampled problem instances (start location, goal location, and goal possibilities), and over 50 runs per problem instance ∆T is the number of time steps separating two consecutive human inputs The robot's mission time is Tmax = 30 time steps We selected γ h = 1 5, corresponding to relatively noisy human inputs and making the problem more difficult to solve for the robot 
Computation times for Goal Only and Path Preference methods on Map 1 (fig 5a),Map 2 (fig 5b), and Map 3 (fig 5c),averaged over 100 runs with randomly sampled problem instances The 95 % confidence interval is provided with the mean We evaluate computation time at the first iteration of each run (where the search depth takes on its highest value Tmax) 

abstract

Robots that can effectively understand human intentions from actions are crucial for successful human-robot collaboration  In this work, we address the challenge of a robot navigating towards an unknown goal while also accounting for a human's preference for a particular path in the presence of obstacles 
This problem is particularly challenging when both the goal and path preference are unknown a priori  To overcome this challenge, we propose a method for encoding and inferring path preference online using a partitioning of the space into polytopes  Our approach enables joint inference over the goal and path preference using a stochastic observation model for the human 
We evaluate our method on an unknown-goal navigation problem with sparse human interventions, and find that it outperforms baseline approaches as the human's inputs become increasingly sparse  We find that the time required to update the robot's belief does not increase with the complexity of the environment, which makes our method suitable for online applications 

INTRODUCTION

Collaboration between humans and robots has become increasingly important and one key aspect of this collaboration is the ability for robots to adapt to human decisions  In many scenarios, such as a robot navigating through a busy room to deliver an item, it is important for the robot to take into account human preferences 
For instance, humans may prefer a specific path that would allow their colleagues to notice the item being delivered, but this preference may change dynamically based on various factors such as changes in the environment or unforeseen circumstances  While some preferences can be incorporated into the path-planning process, accommodating dynamic user preferences in real-time remains challenging 
In this paper, we propose a way to enable robots to adapt to human preferences dynamically by leveraging real-time feedback to inform decision-making  In this work, we tackle the problem of robot navigation in which the robot cannot observe the goal or the preferred path to the goal, but must make navigation decisions that are influenced by humans through recommended actions 
Prior work has explored how to adapt to a human's preference through feedback, but such approaches often require a high level of intervention, which can be time-consuming and impractical in real-world scenarios  To optimize the use of human input and quickly infer the human's preference, Fig  : An autonomous robot navigates in a simulated classroom towards a goal location (pink circle) 
At the start of its mission, it receives direction indications (arrows) from a human that indicate which path it should take to get to the goal  In this scenario, the human wants the robot to go around the desks on the right side of the classroom  A robot that does not reason over path preferences (green) will take the shortest path to the goal regardless of the human's input 
Our method (blue) infers the human's path preference from these indications and adapts to their recommendations  we propose an approach that leverages probabilistic representations of human preference and incorporates real-time feedback  Previous research by Bajcsy et al  considered an online adaptation problem in a manipulation task, where the person can apply forces to the robot to indicate their preferences 
By allowing the robot to continue its task while taking into account a probabilistic representation of human preference, their approach does not require frequent inputs  Building on this idea, we adopt a similar approach to adapt to a human's preference in the context of a robot autonomously navigating through a known environment, such as a cluttered office space 
Specifically, we focus on allowing the human to influence the robot's trajectory with respect to obstacles, by providing guidance on preferred routes or paths, while the robot continues to execute its task  Paths can be represented using homotopy classes   However, homotopies can pose computational challenges when used to encode and infer human preferences 
When the robot maintains a belief over homotopy classes, the inference problem can become exponentially complex with the number of obstacles in the space  Additionally, when the goal is unknown, the number of variables increases with the number of candidate destinations  This complexity can render the decision-making problem intractable 
Our solution is to encode path preference based on a partitioning of the environment into polytopes   This representation allows path preferences to be expressed as sets of preferred transitions between adjacent polytopes  Paths belonging to different homotopy classes correspond to different sequences of transitions 
By leveraging conditional independence assumptions, we can make the Bayesian inference problem tractable  These assumptions exploit the fact that human actions provide information about the path in a piece-wise manner  For example, indicating a preference for navigating around a particular obstacle only provides information about the local area and not the entire path 
Finally, after updating its belief representation over the human's preference, the robot can adapt to indications by replanning online  Our contributions are as follows  • We formulate the human-robot collaboration problem as a Partially Observable Markov Decision Process (POMDP) where both the goal of the task and the human's path preference are unknown random variables 
• We propose an encoding of a human's path preference using a partitioning of the environment into polytopes, along with conditional independence assumptions that make the Bayesian inference problem tractable to infer the task goal and path preference online  • Through simulations in two environments of different sizes and complexity, we show that our method is effective for solving problems where the robot must reach a goal that is unknown a-priori while simultaneously adapting to a human's indications 
Our method shows higher success rates compared to baseline approaches when the human inputs are sparse  Our approach enables a robot to make effective navigation decisions in collaboration with a human, even when the goal and path preference are not known in advance, and with minimal human input  In recent years, there has been a growing interest in shared autonomy and interactive systems, where humans and robots work together to accomplish tasks 
Several approaches have been proposed to address the challenge of enabling effective collaboration between human and robot agents while still achieving high task performance  Losey et al  and Jeon, Losey, and Sadigh propose a framework where a human operator is given control of a task-relevant latent action space while an autonomous system handles the rest 
Dragan and Srinivasa present a formalism for arbitrating between a user's input and a robot's policy when both human and robot share control of the same action space  Cognetti et al  [7] provide a method for real-time modifications of a path,       Fig  : We model the intent inference problem with the above diagram 
At each step in time, the robot receives an observation ot from the human conditioned on its current location st, the intended goal g, and the human's path preference θ  The robot updates its belief over g and θ and transitions to a next location st+1  while Hagenow et al  present a method that allows an outside agent to modify key robot state variables and blends the changes with the original control 
However, a common challenge of these approaches is the high level of intervention required from humans  Best and Fitch propose a method for predicting an agent's intended trajectory from observations  Rather than maintaining a belief over the agent's future path, they infer the agent's intended goal among a set of candidate locations at the boundary of the space 
This approach provides information on where the agent is heading and generates a distribution of candidate future trajectories for the agent  Inferring the goal of the task among a discrete set of candidates is also relevant to the area of shared autonomy  Javdani, Srinivasa, and Bagnell propose a formalism for shared control of a robotic arm, where the robot must assist the human in picking up an object but needs to infer which object the human has chosen from joystick inputs 
Planning with homotopy class constraints is useful in problems where the robot's requirements are given with respect to obstacles, and Yi, Goodrich, and Seppi consider topological constraints provided by human operators  Bhattacharya propose an efficient algorithm for solving pathplanning problems under homotopic constraints 
However, the number of homotopy classes for a given problem can be infinite, and as the robot changes location and updates its representation of the world, carrying out inference over homotopy classes in a dynamic environment requires recomputing the set of homotopies at every iteration, making the belief update challenging 
Prior work has addressed the challenge of shared autonomy by considering how robots can infer a human's intended goal, or how they can infer the preferred path to a goal  However, we argue that inferring the goal and the path as separate problems can lead to over-confidence in incorrect beliefs about the user's preferences 
To illustrate this point, consider the following scenario: a robot and a human are collaborating to move an object from one end of a room to Fig  : Using the hyperplanes composing the H-representation of each obstacle, we construct a hyperplane arrangement of the obstacle-free space (a)  We define the human's preference for the robot's one step action choices as the posterior distribution (given all human input up to that point) over transitions from the current to the neighboring polytopes, i e  edges on the graph 
Each time the robot transitions to a new polytope, the set of neighbor polytopes and the distribution over human preferences are updated  another, but there is an obstacle in the way  The human would like the robot to take a path around the obstacle on the left, even though the goal is on the right  If the robot only infers the goal from the human's inputs, it may incorrectly assume that the goal is on the right, and become over-confident in this belief 
On the other hand, if the robot only infers the preferred path, it may mistakenly assume that the goal is on the left, leading to a failure in completing the task  To overcome these challenges, our work proposes a joint inference approach that considers both the human's intended goal and their preferred path to that goal 
Specifically, we model the human's preference over different homotopy classes and leverage a conditional independence assumption to provide a tractable solution  In our approach, we assume that the human's inputs are noisily rational conditioned on both the goal and the preference  By jointly inferring the goal and path preference, we can avoid over-confidence in incorrect beliefs about the user's preferences, leading to improved system performance 
We consider the problem of robot navigation in a known environment to an unknown destination, where a human can intervene and provide a heading direction to the robot using a joystick or force cues  The human also has a preference on which path the robot should take with respect to obstacles, and our objective is for the robot to understand the human's intentions and execute the task with minimal interventions 
Let g be a discrete random variable denoting the goal of the task, belonging to a set of candidates Ω g , and let θ be a discrete-valued random variable representing the human's path preference, belonging to a set of possible preferences Θ  The physical location of the robot at time index t is denoted by s t ∈ R 2 , and the robot's action at time index t, belonging to some action space A, is denoted by a t  
The transition model T (s t+1 | s t , a t ) is deterministic, meaning the robot has full control over its future location  At any time step, the human may provide an observation to the robot  When the human intervenes, the robot receives a direction (heading angle) that can be mapped to a future location in space 
More specifically, we map the direction to an intended location, which is the resulting robot location after advancing in the indicated direction for one time step  For simplicity, we consider that the robot directly makes an observation o t of the location indicated by the human  We assume that the robot has a stochastic observation model for the human P (o t | s t , g, θ) that is conditioned on both the goal of the task g and the human's preferred path θ 
We further assume that having chosen a goal and path preference, the human takes actions to noisily minimize a cost function C g,θ that measures the cost of moving from the robot's current location to the goal along the preferred path  For example, C g,θ (s t , o t ) can be the length of the shortest path from location s t to the goal g after taking a first step to o t , and constrained by path preference θ 
We use C g,θ to induce a probability distribution over observations, given by: where γ h is a hyperparameter that designates the rationality coefficient  This model assumes the human will pick the lowest cost action with the highest probability and the likelihood of an action decreases exponentially with the increase in cost  
Our inclusion of the path preference θ sets our approach apart from   The model is shown in fig  represented as a Bayesian Network 

Inference

At each time step where the human provides an observation, the posterior P (g, θ) is given through the Bayesian update We note that the number of Bayesian updates required at each iteration to update the belief is equal to the cardinality of Ω g × Θ  In addition, each Bayesian update involves computing C g,θ (  
,   ) in eq  ( ), which involves solving an optimization problem (such as a shortest path problem)  In section IV, we propose a specific encoding of preference θ for resolving eq  ( ), while ensuring the number of computations of the cost C g,θ ( ,  ) per update does not grow exponentially with the number of obstacles 

Decision Making

We consider a navigation problem where the robot receives reward according to the model R(s t , g, θ, a t )  We wish to find the optimal policy π that maximizes the expected discounted sum of future rewards, with discount factor γ  The above problem is a Partially Observable Markov Decision Process (POMDP)  
In this section, we propose an encoding of human's path preference θ for computing the posterior in eq  ( )  Devifrom the concept of homotopy classes, we define the preference according to a partitioning of the environment into polytopes, as shown in fig  , creating a hyperplane arrangement of the space 
Hyperplane arrangements have been used by Vincent and Schwager in the context of Neural Network verification  In our setting, we leverage this representation to define path preferences as preferred transitions between adjacent regions of the space 

Hyperplane Arrangement

We assume a two-dimensional environment composed of m polytopic obstacles, each defined by their half-space representation (H-representation) where A i ∈ R di×2 and b i ∈ R di , and where d i is the number of edges (hyperplanes) composing polytope i  Let n = i d i be the total number of hyperplanes  We leverage each obstacle's H-representation to construct a hyperplane arrangement of the environment as shown in fig 
 e  a partitioning of the space into polytopes  More specifically, each location in space belongs to a polytope j for which we can write an H-representation of the form where α j i ∈ {−1, 1} di is a vector specific to polytope j and obstacle i corresponding to the relative position of any point in the set with respect to each hyperplane in O i  
Fig  : Intent inference model in a hyperplane arrangement of the obstacle free space  We spatially decompose the preference θ into a set of preferred neighboring polytopes per region of the space  Within each polytope j, the human preference pj is a discrete distribution over the preferred neighbor in N (j) 
We assume that for a location st belonging to polytope j, and given goal g and preference pj, the observation ot and any other preference p i,i =j are conditionally independent  Concatenating elements from each obstacle's Hrepresentation, we can write polytope j's H-representation as where Some of the constraints in eq  ( ) (corresponding to rows of A, b and α j ) are redundant, i e  the set P j does not change upon their removal 
We can further reduce the Hrepresentation of a polytope to include only non-redundant constraints  By removing the rows corresponding to redundant constraints, we obtain new matrices A j e , b j e and α j e such that we can write the polytope's reduced H-representation as The non-redundant constraints correspond to edges of the polytope 
In other words, as the robot continually moves in space, the first hyperplane that it will cross upon exiting the polytope will correspond to one of the polytope's nonredundant constraints  Vincent and Schwager outline an iterative method for removing redundant constraints by solving n linear programs 
We use this method in practice for computing α j e for each polytope  We can now characterize each polytope by a vector α j e ∈ {−1, 1} n j e , where n j e ≤ n is the number of essential constraints of the polytope  The polytopes P j partition the environment into a hyperplane arrangement 

Path Preference

In this section, we provide a definition of preference θ according to a graphical representation of the environment based on the hyperplane arrangement  Under this representation, a path preference corresponds to a set of preferred transitions  In other words, for each polytope in the space, the human will have a preference to which neighboring polytope they wish to transition 
Let G := (V, E) be an undirected graph, where vertices are obstacle-free polytopes, and edges connect two adjacent polytopes  Each polytope is described by a unique vector α j as defined in eq  ( )  Two polytopes are adjacent if they share non-redundant constraints (rows in eq  ( )) corresponding to the same hyperplane (i e  they are on opposite sides of the hyperplane) 
Let N (v) be the set of neighbors of a vertex v  For each vertex, we denote p v the discrete-valued random variable describing which edge in N (v) the human intends to transition to  Using this formalism, we define a path preference as the set of preferred transitions over all nodes in the graph, Let m θ = v∈V |N (v)| be the cardinality of Θ, and m g = |Ω g | the number of possible goals 
A priori, the number of Bayesian updates required to update the belief at every iteration should be m θ × m g   Now, let us assume the conditional independence relationships described by the new problem diagram in fig    More specifically, we introduce the assumption that conditioned on a robot location s t , the goal g, and the preference for the corresponding vertex p v in the graph, the observation o t and the preference for any other vertex are conditionally independent 
In other words, the observations the human provides can be defined conditioned only on the robot location, the goal, and the human's preference for its current vertex p v   By introducing this assumption, each update step only requires updating the joint (p v , g), reducing the number of cost computations to |N (v)| × m g  
We can notice that by introducing this assumption, we removed the direct relationship between the number of polytopes in the environment and the complexity of the Bayesian update in eq  ( )  In practice, components of θ are not mutually independent  For example, if the human preference at a vertex v 1 is
, it is unlikely that the human will also prefer p v2 = (v 2 , v 1 ) (turning back)  We can improve our model by assuming a dependent relationship between preferences for adjacent edges, which does not significantly increase the complexity of the inference problem  An interesting property of our encoding is that any two paths that belong to different homotopy classes will cross different sequences of polytopes, i e  they correspond to a different sequence of edges on G 
This can be proved by contradiction  Let us suppose that two continuous trajectories ξ 1 and ξ 2 , with the same start and end points and that do not intersect any obstacle, traverse the same regions in G in the same order  From the construction of the hyperplane arrangement, each polytope that the paths traverse through is obstacle-free 
Therefore, within each polytope, there is no obstacle in the area located in between the portions of ξ 1 and ξ 2 that belong to the region  A smooth transformation of ξ 1 into ξ 2 can be obtained by transforming each portion of ξ 1 belonging to the polytopes it intersects into the corresponding portion of ξ 2 for the same polytopes, where the extremities of the trajectory portions are connected to one another along the polytope's edges (where the same edge is crossed by both paths) 
Along this transformation, the paths do not intersect any obstacle, and therefore ξ 1 and ξ 2 belong to the same homotopy class 

EXPERIMENTS

We evaluate our model on a simulated navigation task where the robot must reach a goal that is unknown a priori while respecting the path preferences indicated by a human  The robot navigates in a grid world containing obstacles  The transition model is deterministic: the robot selects an adjacent location on the grid to reach at the next time step 
The robot is also allowed to take diagonal actions  Each location s t in the map can be mapped to a vertex v t ∈ G  Therefore, the actions leading to locations mapped to different vertices correspond to edges on the graph  We note f (s t , a t ) the edge crossed by taking action a t from location s t  
The robot is given a mission time limit T max for reaching the goal  In this problem, we assume that the human selects actions to noisily minimize a cost function C g,θ , where θ is defined as per eq  ( ), corresponding to the length of the shortest path to the goal constrained by the preference (where the robot is only allowed to make transitions on G along preferred edges) 
More specifically, where δ(s t , g | o t , p vt ) designates the length of the shortest path from s t to g passing by o t and constrained by preference p vt   This is a slight variant of the cost function proposed by Best and Fitch , where we add in a conditioning on the path preference  We compute costs by running the A path planning algorithm on the environment maps (grid worlds with diagonal actions) and impose preference constraints by pruning invalid transitions from the search tree 
Reward model  At each step in time, the robot receives a reward which is a sum of three components: a goal-specific reward a preference-specific reward or penalty We compute solutions to the POMDP defined in section III-B with the online solver POMCP , and with the particularity that within the rollouts, the robot does not expect to collect human inputs 
Each time a solution is computed, the robot takes an action and may receive an observation  If it does, it updates its belief distribution over the unknown problem variables and resolves the POMDP over a receding horizon 

Baselines

• Goal only  The robot solves the POMDP while ignoring the effects of path preference  Similarly to , we assume the human is taking action to minimize a goaldependent cost C g (s t , o t ) = δ(s t , g | o t ), where the conditioning on the preference is removed  We also omit the path preference's contribution to the reward R pref  
• Compliant  The robot complies with the human input, but does not take an initiative  If the user stops providing information, the robot continues in the last direction indicated for 5 time steps (conserving its momentum), then stops  • Blended  We designed an arbitration function to decide between our proposed policy (accounting for path preferences) and the user's recommendation when the robot receives inputs 
Our metric to evaluate confidence in the robot's prediction for the purpose of arbitration is the entropy of the intention distribution H(g, p i ), where p i denotes the preferred neighbor for the current region  Because our representation of the world is discrete, the arbitration is given by a step function 
Denoted by U , the action corresponding to the human's input, and P , the robot's prediction for the optimal action, we write the policy where we chose h = 1 6 as the confidence threshold 

Results

When evaluating the algorithm, we consider that a run is successful if the robot reached the goal within its allocated mission time T max and only made transitions between graph vertices corresponding to the human's preferences  We vary the time delay between human inputs, from constant guidance (∆ T = 1) to only a single observation (∆ T ≥ T max ) 
Success rates  Table I reports the success rates for experiments conducted over six randomly sampled problem instances and 50 runs per instance in Map 1 (fig  )  When the human provides inputs at every iteration, the compliant policy shows the highest success rates  However, as ∆ T increases, the compliant robot is not able to accomplish the task within the allotted time as it does not receive sufficient inputs to do so, and performance decreases compared to the autonomous baselines 
We find that in these runs, accounting for path preference consistently improves performance compared with the goal-only baseline  Results also show that blending the user's input with the robot's policy (Path Preference + Blend) when the human provides information leads to improved performance  Belief entropy 
Figure shows a challenging problem instance where the directions the human provides do not align directly with the shortest path to the goal  By ignoring the effects of preferences in the problem model (goal only), the robot quickly infers from observations that the upper left goal is less likely than others (P (g) drops) 
The strong decrease in entropy shows that the robot becomes overconfident in this prediction  Overconfidence in an incorrect goal will prevent the agent from finding the correct goal once the human's indications directly align with it, as it needs to correct for the wrong predictions, as shown in the path realization (fig 
)  In this realization, the goal-only method (green robot) fails to search the upper left area within the allotted time  By accounting for path preferences in its model, the blue robot's entropy over the goal distribution decreases more steadily, allowing for it to leverage the human's latest observations and reach the goal successfully 
shows an over-confident prediction (shown by the strong reduction in belief entropy) that the correct goal is less likely, making it more difficult to reach the correct goal compared to a method that accounts for path preference  Computation time  In table II we provide the time required to solve the POMDP, and the time required to update the robot's belief as it receives new observations 
We compute solutions on three maps: a simple 10 × 10 grid world with 8 polytopes (fig  ), a 10 × 10 grid world with 56 polytopes (fig  ), and a 20×20 grid world with 73 polytopes (fig  )  The latter environment being larger, we increase the mission time and the depth of the search tree in POMCP from T max = 30 (Map 1 and Map 2) to T max = 60 (Map 3) 
We do not notice an increase in the time required to update the robot's belief with an increase in problem complexity, which is consistent with our observation that the complexity of the Bayesian update should not increase with the number of obstacles or polytopes  On the contrary, the belief update time on Map 2 and Map 3, containing more obstacles, is reduced compared to the first map 
More obstacles result in fewer iterations when solving the constrained shortest path problem with A   Adding constraints due to the obstacles and polytopes reduces the size of the A search tree  C  Limitations Simulation environments  In our simulations, we hardcoded the preference policy over the maps (e g  in Map 1, go around the table counter-clockwise) 
We randomly sampled problem instances (start and goal locations, and goal options) to reduce the bias introduced by these preference choices  To best evaluate and compare the different approaches, it would be best to sample preferences among a distribution of preferences chosen by a human (for example, from benchmarks resulting from a collection of data) 
Creating such a benchmark is an interesting direction for future work  Hyperplane arrangement construction  The main limitation of our approach is that the size and geometry of each polytope depends strongly on the geometry of the obstacles, as seen in fig    Because of this, the robot can make predictions over preferences that are too refined compared with the topology of the environment 
A direct consequence is that when the size of the polytopes is small, the information provided by the human can be incorrectly interpreted as a preference on the robot's immediate action  Our method can be improved by changing the structure of the hyperplane arrangement so that it relies on the topology of the environment, but does not vary strongly with the geometry of the features in the environment 
For this purpose, topometric maps and region construction algorithms are promising directions  We presented an approach for encoding and inferring a human's path preference in an environment with obstacles  By leveraging a partitioning of the space into polytopes and a stochastic observation model, our method allows for joint inference over the goal and path preference even when both are unknown a-priori 
Our experiments on an unknown-goal navigation problem with sparse human interventions demonstrate the effectiveness of our approach and its suitability for online applications  The time required to update the robot's belief does not increase with the complexity of the environment, which further highlights the practicality of our method 



ground_truth: Environmental fluctuation and uncertainty control the reliance of artificial organisms on plasticity.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: What is the effect of accounting for path preference on the robot's belief update?

Read the following text and answer briefly.

Context 3: Current address: Division of Brain Sciences, Department of Medicine, Imperial College London, London, United Kingdom 
In a variety of species, reduced food intake, and in particular protein or amino acid (AA) restriction, extends lifespan and healthspan  However, the underlying epigenetic and/or transcriptional mechanisms are largely unknown, and dissection of specific pathways in cultured cells may contribute to filling this gap  We have previously shown that, in mammalian cells, deprivation of essential AAs (methionine/cysteine or tyrosine) leads to the transcriptional reactivation of integrated silenced transgenes, including plasmid and retroviral vectors and latent HIV-1 provirus, by a process involving epigenetic chromatic remodeling and histone acetylation  Here we show that the deprivation of methionine/cysteine also leads to the transcriptional upregulation of endogenous retroviruses, suggesting that essential AA starvation affects the expression not only of exogenous non-native DNA sequences, but also of endogenous anciently-integrated and silenced parasitic elements of the genome  Moreover, we show that the transgene reactivation response is highly conserved in different mammalian cell types, and it is reproducible with deprivation of most essential AAs  The General Control Non-derepressible 2 (GCN2) kinase and the downstream integrated stress response represent the best candidates mediating this process; however, by pharmacological approaches, RNA interference and genomic editing, we demonstrate that they are not implicated  Instead, the response requires MEK/ERK and/or JNK activity and is reproduced by ribosomal inhibitors, suggesting that it is triggered by a novel nutrient-sensing and signaling pathway, initiated by translational block at the ribosome, and independent of mTOR and GCN2  Overall, these findings point to a general transcriptional response to essential AA deprivation, which affects the expression of non-native genomic sequences, with relevant implications for the epigenetic/transcriptional effects of AA restriction in health and disease 
Copyright: © 2018 De Vito et al  This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited 
Data Availability: All relevant data are within the paper and its Supporting Information files  RNAseq data are available in the ArrayExpress database under the accession number E-MTAB-6452 
Funding: This study was funded by the Ajinomoto Innovation Alliance Program, (AIAP; https://www ajinomoto com/en/rd/AIAP/index html#aiap) (to M V S and D G), which is a joint research initiative of Ajinomoto Co , Inc , Japan  One of the authors [M B ] is an employee of Ajinomoto Co , and his specific roles are articulated in the ‘author contributions’ section  The commercial funder provided support in the form of salary for author [M B ] and some of the necessary research materials (medium for cell culture), but did not have any additional role in the study design, data collection and analysis, or preparation of the manuscript, and the authors had unrestricted access to the data  Due to a confidentiality agreement, the commercial funder participated only in the decision to publish the data obtained during the study, without any restriction 
Competing interests: This study was funded by Ajinomoto Co , Inc , Japan and one of the authors [M B ] is an employee of this commercial funder  No other employment or consultancy relationships exist with the commercial funder, and no patents, products in development, or marketed products result from this study  The authors declare that no competing interests exist and that the commercial affiliation of one of the authors does not alter the adherence of authors to all PLOS ONE policies on sharing data and materials 
In animals, excessive, insufficient, or imbalanced nutrient availability is known to strongly impact on phenotype and health, both short and long-term, and across generations [1, 2]  In particular, studies in yeast, animal models and humans have shown that reduced food intake, reducing either overall calories, or only sugars, proteins, or even single amino acids (AA), such as Methionine (Met), may extend lifespan and healthspan, and reduce the risk of cancer and other age-related diseases [3–9]  In addition, fasting or specific AA deprivation have shown potential therapeutic applications, owing to their ability to directly reduce the growth of some tumor types [10, 11], sensitize cancer cells to chemo- or immunotherapy [12, 13], and allow efficient hematopoietic stem cell engraftment   However, little is known about the specific processes and molecular mechanisms mediating the roles of nutrient restriction in human health and longevity 
A properly balanced diet in metazoans contains optimal amounts of a subset of AA, which cannot be synthetized de novo and are therefore named essential amino acids (EAAs)  In humans these include Met, Histidine (His), Isoleucine (Ile), Leucine (Leu), Lysine (Lys), Phenylalanine (Phe), Threonine (Thr), Tryptophan (Trp), and Valine (Val), while a few others are considered as semi-essential, such as Glutamine (Gln) and Tyrosine (Tyr) [15, 16]  Consistently, EAA deprivation triggers a cell-autonomous adaptive response, characterized by extensive metabolic and gene expression modifications, implementing biosynthetic, catabolic, and plasma membrane transport processes, aimed at reconstituting the full AA complement [17, 18]  The best known and conserved pathways responding to AA deprivation are triggered by mechanistic Target of Rapamycin Complex 1 (mTORC1) and General amino acid Control Non-derepressible 2 (GCN2) protein kinases [15, 19, 20]  Activation of mTORC1 requires in particular the presence of Gln, Arg and Leu, but also Met , which activate the kinase through sensors mainly acting upstream of Rag GTPases at lysosomal membranes   In turn, mTORC1 promotes cell growth, proliferation and anabolism upon activation, and translational attenuation and autophagy upon inhibition [19, 20] 
By contrast, GCN2 is activated by deprivation of any individual EAA, by means of its histidyl-tRNA synthetase-related domain, which binds uncharged tRNAs accumulating during AA limitation [23, 24]  Upon activation, GCN2 phosphorylates and inhibits its only known downstream target, namely the eukaryotic Initiation Factor 2 α (eIF2α), thereby initiating the Integrated Stress Response (ISR)  This leads to attenuation of general translation, and induction of a transcriptional/translational program, aimed at increasing stress resistance and restoring cell homeostasis, by upregulating a specific subset of genes, including Activating Transcription Factor 4 (ATF4) and C/EBP-Homologous Protein (CHOP) [25–27]  Thus, inhibition of mTORC1 and activation of GCN2 by AA restriction cooperate to attenuate general translation at the initiation step, increase catabolism and turnover, and enhance stress resistance to promote adaptation   However, how these processes eventually induce protective mechanisms against the alterations associated with aging, which include pervasive epigenetic and transcriptional changes [28, 29], remains largely unknown 
We previously reported the unexpected observation that prolonged deprivation of either Tyr, or of both Methionine and Cysteine (Met/Cys), triggers the selective and reversible reactivation of exogenous transcriptional units, including plasmids, retroviral vectors and proviruses, integrated into the genome and transcriptionally repressed by defensive mechanisms against non-native DNA sequences [30, 31]  This phenomenon was observed both in HeLa epithelial and ACH-2 lymphocytic human cells, and was independent of the transgene or provirus (Ocular Albinism type 1, OA1; Green Fluorescent Protein, GFP; Lysosomal-Associated Membrane Protein 1, LAMP1; Human Immunodeficiency Virus-1, HIV-1), or of the exogenous promoter driving their transcription, either viral (cytomegalovirus, CMV; Long Terminal Repeat, LTR) or human (Phospho-Glycerate Kinase 1, PGK1; Elongation Factor-1α, EF-1α)   Furthermore, this transgene reactivation response was not reproduced by serum starvation, activation of p38, or pharmacological inhibitors of mTOR (PP242 or rapamycin), sirtuins and DNA methylation  By contrast, it was induced by pan histone deacetylase (HDAC) inhibitors, and by selective inhibitors of class II HDACs   Consistently, we found that the mechanism responsible involves epigenetic modifications at the transgene promoter, including reduced nucleosome occupancy and increased histone acetylation, and is mediated in part by reduced expression of a class II HDAC, namely HDAC4  
These findings indicate that AA deprivation induces a specific epigenetic and transcriptional response, affecting the expression of newly-integrated exogenous transgenes and proviruses, and suggesting that endogenous sequences sharing similar structural and functional features may represent a transcriptional target as well [30, 31]  In particular, transposable elements, such as LTR-retrotransposons (or endogenous retroviruses, ERVs), are genomic “parasites” anciently-integrated into the genome, and silenced by epigenetic mechanisms of mammalian cells against the spreading of mobile elements, eventually becoming "endogenized" during evolution [32, 33]  This raises the question of whether their expression is also sensitive to AA restriction  In addition, it remains unclear whether or not the transgene reactivation response is related to specific AA deprivations, and most importantly which is the AA sensing/signaling pathway involved, in particular whether the GCN2 kinase is implicated  Thus, here we used the reactivation of silenced transgenes in cultured cells, as a model to investigate a novel molecular pathway induced by imbalanced EAA starvation, implicated in the epigenetic/transcriptional regulation of exogenous non-native DNA sequences and possibly of other endogenous anciently-integrated genomic elements 
HeLa human epithelial carcinoma, HepG2 human hepatocellular carcinoma and C2C12 mouse skeletal muscle cells were maintained in DMEM containing glutaMAX (Invitrogen) and supplemented with 10% FBS (Sigma), 100 U/ml penicillin G (Invitrogen), 100 mg/ml streptomycin (Invitrogen), at 37°C in a 5% CO2 humidified atmosphere  Cell lines carrying integrated and partially silenced transgenes were also maintained in 600–1000 μg/ml G418 
The C2C12 cell line was provided by ATCC  HeLa and HepG2 cells were obtained by Drs  F  Blasi and G  Tonon at San Raffaele Scientific Institute, Milan, Italy, respectively, and were authenticated by Short Tandem Repeat (STR) profiling, using the Cell ID System kit (Promega), according to the manufacturer’s instructions  Briefly, STR-based multiplex PCR was carried out in a final volume of 25 μL/reaction, including 5 μL Cell ID Enzyme Mix 5X, 2 5 μL Cell ID Primer Mix 10X and 3 ng of template DNA  The thermal cycling conditions were: 1 cycle at 96°C for 2 min, followed by 32 cycles at 94°C for 30 sec, 62°C for 90 sec, and 72°C for 90 sec, and 1 cycle at 60°C for 45 sec  The following STR loci were amplified: AMEL, CSF1PO, D13S317, D16S539, D21S11, D5S818, D7S820, TH01, TPOX, vWA  Fragment length analysis of STR-PCR products was performed by Eurofins Genomics, using standard procedures of capillary electrophoresis on the Applied Biosystems 3130 XL sequencing machine, and assessment of the STR profile was performed at the online STR matching analysis service provided at http://www dsmz de/fp/cgi-bin/str html 
Stable cell clones, expressing myc-tagged human OA1 (GPR143) or GFP transcripts, were generated using pcDNA3 1/OA1myc-His or pcDNA3 1/EGFP vectors   Briefly, HeLa, HepG2 and C2C12 cells were transfected using FuGENE 6 (Roche) and selected with 800, 1000, and 650 μg/ml of G418 (Sigma), respectively, which was maintained thereafter to avoid loss of plasmid integration  G418-resistant clones were isolated and analyzed for protein expression by epifluorescence and/or immunoblotting 
Full DMEM-based medium, carrying the entire AA complement, and media deprived of Met/Cys (both AAs), Met (only), Cys (only), Alanine (Ala), Thr, Gln, Val, Leu, Tyr, Trp, Lys and His were prepared using the Nutrition free DMEM (cat #09077–05, from Nacalai Tesque, Inc , Kyoto, Japan), by adding Glucose, NaHCO3, and either all 20 AAs (for full medium) or 18–19 AAs only (for deprivations of two-one AAs)  Single AAs, Glucose, and NaHCO3 were from Sigma  Further details and amounts utilized are indicated in S1 Table  All media were supplemented with 10% dialyzed FBS (Invitrogen), 100 U/ml penicillin G (Invitrogen), 100 mg/ml streptomycin (Invitrogen), and G418 as required  HBSS was from Invitrogen  Cells were seeded at 10–30% of confluency; cells to be starved for 48 h were plated 2–3 times more confluent compared to the control  The following day, cells were washed and cultured in the appropriate medium, with or without EAA, for 24–48 h 
L-Histidinol (HisOH), PP242, Integrated Stress Response Inhibitor (ISRIB), SP600125, Cycloheximide (CHX) were from Sigma; Salubrinal was from Tocris Bioscience; U0126 was from Promega  Drugs were used at the following final concentrations: HisOH at 4–16 mM; PP242 at 1–3 μM; ISRIB at 100 nM; SP600125 at 20 μM in HepG2 cells and 50 μM in HeLa cells; Cycloheximide (CHX) at 50 ug/ml in HepG2 cells and 100 ug/ml in HeLa cells; Salubrinal at 75 μM; U0126 at 50 μM  Vehicle was used as mock control  Treatments with drugs to be tested for their ability to inhibit transgene reactivation (ISRIB, SP600125 and U0126) were initiated 1h before the subsequent addition of L-Histidinol (ISRIB) or the subsequent depletion of Met/Cys (SP600125 and U0126) 
Total RNA was purified using the RNeasy Mini kit (Qiagen), according to manufacturer’s instructions  RNA concentration was determined by Nanodrop 8000 Spectrophotometer (Thermo Scientific)  Equal amount (1 μg) of RNA from HeLa, HepG2 and C2C12 cells was reverse transcribed using the SuperScript First-Strand Synthesis System for RT-PCR (Invitrogen) using oligo-dT as primers, and diluted to 5 ng/μl  The cDNA (2 μl) was amplified by real-time PCR using SYBR green Master Mix on a Light Cycler 480 (Roche), according to manufacturer’s instructions  The thermal cycling conditions were: 1 cycle at 95°C for 5 min, followed by 40–45 cycles at 95° for 20 sec, 56° for 20 sec and 72° for 20 sec  The sequences, efficiencies and annealing temperatures of the primers are provided in S2 Table  Data were analyzed with Microsoft Excel using the formula EtargetΔct target (control-sample) /EreferenceΔct reference (control-sample)   Reference genes for normalizations were ARPC2 (actin-related protein 2/3 complex, subunit 2) for HeLa and HepG2 cells; and Actb (actin beta) for C2C12 cells, unless otherwise indicated 
siRNA (Mission esiRNA, 200 ng/μL; Sigma) against ATF4 and GCN2 were designed against the targeted sequences NM_001675 and NM_001013703, respectively  Cells seeded in 6-well plates were transfected with 1 μg of siRNAs and 5 μL of Lipofectamine 2000 (Invitrogen), following manufacturer’s instructions, at day 1 post-plating for ATF4 and at day 1 and 2 post-plating for GCN2  At day 2 (ATF4) or 3 (GCN2) post-plating, cells were washed and cultured in medium in the absence or presence of HisOH 4 mM for 6 h  siRNAs against RLuc (Sigma), targeting Renilla Luciferase, were used as negative control  For CRISPR/Cas9 experiments, we used the “all-in-one Cas9-reporter” vector, expressing GFP (Sigma), which is characterized by a single vector format including the Cas9 protein expression cassette and gRNA (guide RNA)  GFP is co-expressed from the same mRNA as the Cas9 protein, enabling tracking of transfection efficiency and enrichment of transfected cells by fluorescence activated cell sorting (FACS)  The human U6 promoter drives gRNA expression, and the CMV promoter drives Cas9 and GFP expression  The oligonucleotide sequences for the three gRNAs targeting GCN2 exon 1 or 6 are listed in S2 Table  We transfected HeLa and HepG2 cells with these plasmids individually (one plasmid one guide) and sorted the GFP-positive, transfected cells by FACS  Screening GCN2-KO clones was performed by western blotting  In the case of HepG2-OA1 cells, two rounds of selection were necessary to obtain three GCN2-KO clones by using a guide RNA against exon 1  Compared to the original HepG2-OA1 cell line and to the clone resulting from the first round of selection (185#27), the selected clones E23, F22 and F27 showed a very low amount—if any—of residual GCN2 protein (see results) 
Genomic DNA of HeLa and HepG2 cells was purified using DNeasy Blood and Tissue kit (Qiagen), according to the manufacturer’s instructions  DNA concentration was determined by Nanodrop 8000 Spectrophotometer (Thermo Scientific)  PCR conditions for amplification of GCN2 exon 1 and 6 were as follows: 1 cycle at 94°C for 5 min, followed by 35 cycles at 94°C for 40 sec, 56°C for 40 sec, and 72°C for 40 sec; and a final extension step of 5 min at 72°C  The primer sequences are provided in S2 Table 
For OA1, western immunoblotting was carried out as described   For GCN2, cells were lysed in RIPA buffer, boiled at 95°C for 5 min and resolved on a 7 5% polyacrylamide gel; immunoblotting was then performed following standard procedures  Primary Abs were as follows: anti-human OA1, previously developed by our group in rabbits ; anti-GCN2 (Cell Signaling, Cat  #3302) 
Statistical analyses were performed using Microsoft Excel for Mac (version 15 32, Microsoft) for Student’s t-test; or GraphPad Prism (version 5 0d for Mac, GraphPad Software, Inc ) for one-way analysis of variance (ANOVA), followed by Dunnett’s or Tukey’s multiple comparisons post-tests  T-test was used when only two means, typically sample versus control, were compared, as specified in the figure legends  One way ANOVA was used for multiple comparisons, followed by either a Dunnett’s (to compare every mean to a control mean), or a Tukey’s (to compare every mean with every other mean) post-test, by setting the significance level at 0 05 (95% confidence intervals)  Both tests compare the difference between means to the amount of scatter, quantified using information from all the groups  Specifically, Prism computes the Tukey-Kramer test, allowing unequal sample sizes  P values in Figures are generally referred to comparison between a sample and the control (full medium/mock), and are indicated as follows: *P<0 05, **P<0 01, ***P<0 001  Comparisons not involving the control are similarly indicated, by a horizontal line at the top of the graphs, encompassing the two samples under analysis  Additional details regarding the specific experiments are reported in the Figure Legends 
To examine the expression behavior of genomic repeats upon AA starvation, we performed a transcriptomic analysis taking advantage of an intramural sequencing facility  HeLa-OA1 cells were cultured in normal medium (for 6-30-120 hours) or in absence of Met/Cys (for 6-15-30-72-120 hours)  Total RNA was prepared using Trizol (Sigma) to preserve transcripts of both small and long sizes (from Alu, of about 0 3 kb, to Long Interspersed Nuclear Elements, LINEs, and ERVs, up to 6–8 kb long), DNase treated to avoid contamination of genomic DNA, and processed for NGS sequencing by Ovation RNA-Seq System V2 protocol and HiSeq 2000 apparatus  Raw sequence data (10–20 M reads/sample) were aligned to the human genome (build hg19) with SOAPSplice   Read counts over repeated regions, defined by RepeatMasker track from UCSC genome browser , were obtained using bedtools suite   Normalization factors and read dispersion (d) were estimated with edgeR , variation of abundance during time was analyzed using maSigPro package , fitting with a negative binomial distribution (Θ = 1/d, Q = 0 01), with a cutoff on stepwise regression fit r2 = 0 7  Read counts were transformed to RPKM for visualization purposes  The expression of the OA1 transgene and HDAC4, which are progressively up- and down-regulated during starvation, respectively , were used as internal controls 
For genomic repeat analysis, reads belonging to repetitive elements were classified according to RepeatMasker and assigned to repeat classes (total number in the genome = 21), families (total number in the genome = 56) and finally subfamilies (total number in the genome = 1396), each including a variable number of genomic loci (from a few hundred for endogenous retroviruses, up to several thousand for Alu)  Repeat subfamilies were then clustered according to their expression pattern in starved vs control cells, by maSigPro using default parameters, and repeats classes or families that are significantly enriched in each cluster, compared to all genomic repeats, were identified by applying a Fisher Exact test (using scipy stats, a statistical module of Python)  Alternatively, differentially expressed repeat subfamilies were identified by averaging three time points of starvation (15-30-72 h) and controls  Repeats significantly up- or downregulated (104 and 77, respectively) were selected based on a P value <0 05 (unpaired two-tailed Student’s t-test, assuming equal variance), and analyzed for their class enrichment by a Fisher Exact test as described above 
For gene set enrichment analysis of Met/Cys deprived vs control HeLa cells, differentially expressed genes were selected considering three time points of starvation (15-30-72 h) and controls, based on a P value <0 05 (unpaired two-tailed Student’s t-test, assuming equal variance) and a fold change >2  This led to a total of 2033 differentially expressed genes, 996 upregulated and 1037 downregulated  The enrichment analysis was performed separately for up and down regulated genes, or with all differentially expressed genes together (both), using the KEGG database  The analysis was performed with correction for the background of all expressed genes (about 13600 genes showing an average expression over 3 starvation and 3 control samples of at least 5 counts) and by using default parameters (adjusted P value and q-value cut-off of <0 05 and 0 2, respectively)  Differentially expressed genes were also selected considering all starvation time points, as with genomic repeats, by maSigPro using default parameters, and a fold change of at least 1 5, leading to similar enrichment results (not shown)  RNAseq gene expression data are available in the ArrayExpress database under the accession number E-MTAB-6452 
To provide proof-of-principle that AA starvation may affect the expression of transposable elements, we performed an RNAseq analysis of the previously described HeLa-OA1 cells, carrying an integrated and partially silenced OA1 transgene   Since the reactivation of the transgene by starvation is a progressive phenomenon , we performed a time-course experiment, where each time point represents one biological sample, rather than a biological triplicate of a single time point  To this aim, cells were cultured either in normal medium, or in absence of Met/Cys for different time points (6-15-30-72-120 hours), resulting in the progressive upregulation of the OA1 transgene during starvation (Fig 1A and 1B), consistent with previously published results   The expression of genomic repeats was determined according to RepeatMasker annotation and classification into classes, families, and subfamilies  Repeat species were then subjected to differential expression and enrichment analyses in starved vs control conditions  Out of 1396 annotated repeat subfamilies, 172 species displayed a differential expression profile during starvation 
Fig 1



ground_truth: The belief entropy decreases more steadily.

prompt_front: Now, answer the following question based on the text below, only give me the answer and do not output any other words.

Question: Is the ISR necessary for transgene reactivation?

Read the following text and answer briefly.

Context 3:  Exogenous transgene and endogenous retroviruses are upregulated in Met/Cys-deprived HeLa cells 
(A,B) Exogenous integrated transgene (OA1) mRNA abundance in HeLa-OA1 cells, cultured in Met/Cys-deprived medium for the indicated time points, and analyzed by RNAseq (A), or RT-qPCR (B), compared to full medium  Data represent RPKM (A), or mean ± SD of 2 technical replicates, expressed as fold change vs  control (full medium at 6 h = 1) (B)  (C) Clustering of 172 genomic repeat subfamilies, differentially expressed upon starvation, according to their expression profile  (D) Class distribution of repeat subfamilies belonging to differential expression clusters, compared to all genomic repeat subfamilies (first column)  Class DNA includes DNA transposons; SINE includes Alu; LINE includes L1 an L2; LTR includes endogenous retroviruses and solitary LTRs; Satellite includes centromeric acrosomal and telomeric satellites; Others includes SVA, simple repeats, snRNA, and tRNAs  LTR-retroelements are significantly enriched among repeats that are upregulated upon starvation, while LINEs are significantly enriched among repeats that are downregulated  *P<0 05, ***P<0 001 (Fisher exact test) 
As shown in Fig 1C, the clustering of differentially expressed repeats, according to their expression pattern, reveals profiles comparable to the behavior of the transgene in the same conditions, i e  upregulation upon starvation and no change in regular medium (Cluster 1 and 2)  In particular, Cluster 1 contains sequences that, similarly to the OA1 transgene, are progressively upregulated upon starvation (Fig 1A and 1C) , while Cluster 2 contains sequences that are upregulated at early time points  Interestingly, repeat families that are significantly enriched in these two clusters belong mostly to the group of LTR-retrotransposons, including ERV1, ERVK, ERVL, ERVL-MaLR and other LTR sequences (Fig 1D; S1A and S2A Figs)  By contrast, DNA transposons (such as TcMar-Tigger) and L1 non-LTR retrotransposons are enriched among repeats that are downregulated during starvation, particularly at late time points (Clusters 3 and 4) (Fig 1D; S1A and S2A Figs)  Consistent results were obtained by selecting significantly up- or downregulated genomic repeats (overall 181 species), based on their average expression out of three time points of starvation (15-30-72 h, when the transgene upregulation is more homogeneous) and controls, and on a P value <0 05 (S1B and S2B Figs)  These findings suggest that EAA starvation induces genome-wide effects involving repetitive elements, and that—among major repeat classes—it upregulates in particular the expression of ERVs 
In addition, to obtain a general overview of main gene pathways changing their expression together with the transgene during AA starvation, we performed gene expression and enrichment analyses of regular genes, by considering three time points of starvation (15-30-72 h) and controls  Differentially expressed genes were selected based on a P value <0 05 and a fold change between means of at least 2, and analyzed with the EnrichR tool   As shown in Fig 2 and S1 File, enrichment analyses against the KEGG and Reactome databases reveals a predominance of downregulated pathways, namely ribosome and translation, proteasome, AA metabolism, oxidative phosphorylation and other pathways related to mitochondrial functions, which are affected in Huntington, Alzheimer and Parkinson diseases (http://www genome jp/kegg/pathway html)  In particular, a large fraction of ribosomal protein mRNAs is downregulated upon Met/Cys starvation (Fig 2A and 2C; S1 File), consistent with the notion that their genes–despite being scattered throughout the genome—are coordinately expressed in a variety of conditions   This reduced expression may depend on multiple pathways that control ribosome biogenesis in response to external stimuli, including the downregulation of Myc activity , the downregulation of mTORC1 [42, 44], or possibly the activation of the ISR, as described in yeast   By contrast, upregulated genes show a significant enrichment for transcription and gene expression (Fig 2B)  Similar results were obtained by the Gene Ontology Biological Process (GO-BP) database (S1 File), overall indicating a general downregulation of translation and metabolism, and upregulation of transcription, during the time interval of Met/Cys starvation corresponding to the transgene upregulation 
Fig 2  Gene set enrichment analysis of Met/Cys-deprived HeLa cells 
Differentially expressed genes between three time points of starvation (15-30-72 h) and controls were selected based on a P value <0 05 and a fold change of at least 2, leading to a total of 996 upregulated, and 1037 downregulated genes  The enrichment analysis was performed separately for up and down regulated genes, using the EnrichR tool and the KEGG (A) and REACTOME (B, C) databases  Ranking is based on the combined score provided by EnrichR, and categories are displayed up to 20 items with an Adjusted P value <0 05  No significant categories were found with upregulated genes against the KEGG database  All data are shown in S1 File  The enrichment analysis using all differentially expressed genes together did not reveal any additional enriched process 
To characterize the pathway leading to the reactivation of silenced transgenes, we used HeLa-OA1 and HeLa-GFP cells, as described   In addition, to test cell types relevant for AA metabolism, such as liver and muscle, we generated clones of HepG2 human hepatoma and C2C12 mouse skeletal muscle cells, stably transfected with plasmids for OA1 and GFP transgenes, respectively (HepG2-OA1 and C2C12-GFP cells; endogenous OA1 is not expressed in any of these cell types)  In all cases, the integrated transgenes are under the control of the CMV promoter in the context of a pcDNA3 1 plasmid, are partially silenced, and can be efficiently upregulated by HDAC inhibitors (trichostatin A, TSA; ref   and S3A, S3B and S4A Figs), indicating that their expression is controlled at least in part by epigenetic mechanisms, as previously described  
To establish whether the reactivation response results from the shortage of specific AAs only, such as Met/Cys, or it is triggered by any AA deprivations, we cultured HeLa-OA1, HeLa-GFP, HepG2-OA1 and C2C12-GFP cells for 24–48 hours with a battery of media deprived of EAAs or semi-EAAs, including Met/Cys, Thr, Gln, Val, Leu, Tyr, Trp, Lys, and His  As negative controls, cells were cultured in full medium, carrying the entire AA complement, and in a medium deprived of Ala, a non-essential AA  The expression of the transgene transcript was then evaluated by RT-qPCR  As shown in Fig 3, and in S3C and S4B Figs, most EAA-deficiencies induced reactivation of the OA1 or GFP transgenes in all four cell lines, with the notable exception of Trp deprivation, which consistently resulted in no or minimal reactivation of the transgenes  Indeed, despite some variability, Met/Cys deficiency, but also Thr, Val, Tyr, and His deprivation always gave an efficient response, while Leu, Gln and Lys elicited evident responses in some cases, but not in others  Depletion of Phe gave results comparable to Tyr deprivation, however it significantly altered multiple reference genes used for normalization and therefore was eventually omitted from the analysis (not shown)  Finally, in the above experiments we used a combined Met/Cys deficiency, to avoid the potential sparing of Met by Cys  and for consistency with our previous studies   Nevertheless, the analysis of single Met or Cys starvation, both at the protein and transcript levels, revealed an exclusive role of Met deprivation in transgene reactivation, consistent with the notion that Cys is not an EAA (S3D and S3E Fig) 
Fig 3  EAA deprivation induces reactivation of silent transgenes in HeLa and HepG2 cells 
Relative transgene (OA1) and CHOP mRNA abundance in HeLa-OA1 (A) and HepG2-OA1 (B) cells, cultured in various AA-deprived media for 48 h and 24 h, respectively, compared to full medium  Mean ± SEM of 3 independent experiments  Data are expressed as fold change vs  control (full medium = 1)  *P<0 05, **P<0 01, ***P<0 001 (one way ANOVA, followed by Dunnett’s post-test vs  full medium) 
Collectively, these results indicate that transgene reactivation by EAA starvation is reproducible with most EAAs, shared by different cell types (epithelium, liver, and skeletal muscle), and conserved in different mammalian species (human, mouse) 
mTORC1 inhibition and GCN2 activation trigger the best-known signaling pathways responding to AA starvation   We previously showed that inhibition of mTORC1 is not sufficient to reproduce transgene reactivation in HeLa cells   By contrast, the involvement of GCN2 and the ISR, including the downstream effectors ATF4 and CHOP, has never been tested  In addition, this pathway has been typically assessed in transient assays, lasting for a few hours, which may not be comparable with the prolonged starvation conditions necessary to reactivate the transgene expression (at least 15–24 h)  Thus, we tested whether CHOP expression was upregulated upon incubation of HeLa-OA1, HepG2-OA1 and C2C12-GFP cells in media deprived of different EAAs for 24–48 h 
As shown in Fig 3 and S4B Fig, we found that CHOP expression is increased in all EAA-starvation conditions, but not in the absence of Ala, in all tested cell lines  Similar, yet less pronounced, results were obtained with ATF4, consistent with the notion that activation of this transcription factor is mainly mediated by translational upregulation (not shown) [15, 26]  However, the upregulation of CHOP does not parallel quantitatively that of the transgene, neither appears sufficient to induce it  In fact, CHOP is highly upregulated even upon Trp starvation, which consistently results in no or minimal reactivation of the transgenes (compare CHOP with OA1 or GFP expression; Fig 3 and S4B Fig)  Thus, while the ISR appears widely activated upon EAA starvation, the upregulation of its downstream effector CHOP only partly correlates with transgene reactivation and may not be sufficient to induce it 
The activation of the ISR upon AA starvation suggests that GCN2 may be involved in the transgene reactivation response  Therefore, we tested whether direct pharmacological activation of this kinase is sufficient to trigger the transgene reactivation similarly to starvation  In addition, we used pharmacological inhibitors of mTOR to corroborate previous negative results in HeLa cells  in the other cell lines under study  To this aim, HeLa-OA1 or GFP, HepG2-OA1 and C2C12-GFP cells were cultured in the presence of different concentrations of PP242 (mTOR inhibitor) or L-Histidinol (GCN2 activator, inhibiting tRNAHis charging by histidyl-tRNA synthetase), either alone or in combination for 24 h, compared to Met/Cys-deprived and full medium  As shown in Fig 4 and S5 Fig, while inhibition of mTORC1 consistently leads to minor or no effects, in agreement with previous findings , treatment with L-Histidinol results in efficient reactivation of the transgene in HepG2-OA1 and C2C12-GFP cells, but not in HeLa cells 
Fig 4  mTOR inhibition and GCN2 activation differently affect transgene expression in HeLa and HepG2 cells 
Relative transgene (OA1) and CHOP mRNA abundance in HeLa-OA1 (A) and HepG2-OA1 (B) cells, cultured in Met/Cys-deprived medium, or in the presence of PP242 (mTOR inhibitor; 1–3 μM) or L-Histidinol (HisOH, GCN2 activator; 4–16 mM), either alone or in combination for 24–48 h, compared to full medium  Mean ± SEM of 4 (A) or 3 (B) independent experiments  Data are expressed as fold change vs  control (full medium = 1)  *P<0 05, **P<0 01, ***P<0 001 (one way ANOVA, followed by Dunnett’s post-test vs  full medium)  PP-1 and PP-3, PP242 at 1 and 3 μM, respectively; HisOH-4 and HisOH-16, L-Histidinol at 4 and 16 mM, respectively 
Specifically, L-Histidinol is not effective in HeLa-OA1 and HeLa-GFP cells, either alone or in combination with PP242 (Fig 4A and S5A Fig), or by using different concentrations of the drug, with or without serum (not shown)  In these cells, L-Histidinol appears also unable to trigger the ISR, as indicated by lack of CHOP upregulation, possibly due to their different sensitivity to the drug  These findings are consistent with previous reports, describing the use of L-Histidinol in HeLa cells in conditions of low His concentration in the culture medium , which would resemble AA starvation in our system and therefore may not be applicable  Thus, even though the amount of the amino alcohol was adapted to exceed 20 to 80 times that of the amino acid, as described , HeLa cells may be resistant or able to compensate 
In contrast, in other cell types, L-Histidinol has been utilized in regular DMEM, to mimic the AA response triggered by DMEM lacking His [48, 49]  Consistently, in HepG2-OA1 cells, L-Histidinol is sufficient to elicit extremely high levels of transgene reactivation, and its combination with PP242 results in additive or even synergistic effects, possibly due to an indirect effect of mTOR inhibition on GCN2 activity (Fig 4B) [50, 51]  Similarly, C2C12-GFP cells efficiently reactivate the transgene upon treatment with L-Histidinol, but not PP242 (S5B Fig)  However, differently from HepG2-OA1 cells, simultaneous treatment of C2C12-GFP cells with L-Histidinol and PP242 does not lead to synergistic effects  Consistent with stimulation of the ISR, CHOP and to a minor extent ATF4 are upregulated by L-Histidinol in both cell lines, yet their expression levels show only an incomplete correlation with those of the transgene (Fig 4B, S5B Fig, and not shown) 
The finding that GCN2 activation by L-Histidinol is sufficient to reactivate the transgenes in both HepG2-OA1 and C2C12-GFP cells pointed to this kinase, and to the downstream ISR, as the pathway possibly involved in the EAA starvation response  Thus, we investigated whether the ISR is sufficient to trigger upregulation of the OA1 transgene in HepG2-OA1 cells by pharmacological means  As CHOP expression does not correspond quantitatively and is not sufficient to induce transgene reactivation, we tested the role of the core upstream event of the ISR, namely the phosphorylation of eIF2α , which can be induced by pharmacological treatments, independent of GCN2 (Fig 5A)  To this aim, we used Salubrinal, a specific phosphatase inhibitor that blocks both constitutive and ER stress-induced phosphatase complexes against eIF2α, thereby increasing its phosphorylation   We found that, while the ISR is activated upon Salubrinal treatment, as shown by increased CHOP expression, it does not induce OA1 transgene reactivation (Fig 5B) 
Fig 5  The ISR is neither sufficient nor necessary to induce transgene reactivation in HepG2 cells 
(A) Schematic representation of GCN2 activation by AA starvation, resulting in phosphorylation of eIF2a and initiation of the downstream ISR  In addition to GCN2, the ISR may be activated by other eIF2a kinases (PKR, HRI and PERK; not shown in the picture)  (B) Relative transgene (OA1) and CHOP mRNA abundance in HepG2-OA1 cells treated for 24 h with Salubrinal (a drug that induces the ISR by inhibiting the dephosphorylation of eIF2α; 75 μM), compared to full medium  Mean ± range of two experiments  Data are expressed as fold change vs  control (DMEM = 1)  *P<0 05 (paired two-tailed Student’s t-test vs  control)  (C) Relative transgene (OA1) and CHOP mRNA abundance in HepG2-OA1 cells treated for 6 h with L-Histidinol (HisOH, GCN2 activator; 4 mM), in the absence or presence of ISRIB (a drug that bypasses the phosphorylation of eIF2α, inhibiting triggering of the ISR; 100 nM)  Mean ± range of two experiments  Data are expressed as fold change vs  control (DMEM = 1)  **P<0 01, ***P<0 001 (one way ANOVA, followed by Tukey’s post-test; P values refer to comparisons vs  control, unless otherwise indicated)  (D) Relative transgene (OA1) and ATF4 mRNA abundance in HepG2-OA1 cells transfected with control (CTRL) or anti-ATF4 siRNAs, and incubated in the presence or absence of L-Histidinol (HisOH, GCN2 activator; 4 mM) for 6 h  Mean ± range of two experiments  Data are expressed as fold change vs  control (w/o HisOH = 1, top; control siRNA = 1, bottom)  *P<0 05 (one way ANOVA, followed by Tukey’s post-test; P values refer to comparisons vs  control, unless otherwise indicated) 
To test whether the ISR is necessary to trigger the transgene response to L-Histidinol, we used the chemical compound ISRIB, which inhibits the activation of the ISR, even in the presence of phosphorylated eIF2α, likely by boosting the activity of the guanine-nucleotide exchange factor (GEF) for eIF2α, namely eIF2B [53, 54]  HepG2-OA1 cells were stimulated with L-Histidinol, either in the presence or absence of ISRIB  As shown in Fig 5C, while the expression of CHOP is inhibited by ISRIB, as expected, the reactivation of the OA1 transgene is not affected  In addition, knockdown of the closest eIF2α downstream effector ATF4 by siRNAs does not interfere with the reactivation of the OA1 transgene by L-Histidinol (Fig 5D)  Together, these data suggest that eIF2α phosphorylation and the downstream ISR pathway are neither sufficient nor necessary to induce transgene reactivation 
To definitively establish if GCN2 is necessary to trigger the transgene reactivation response to EAA starvation, we directly suppressed its expression by CRISPR/Cas9-mediated knock-out (KO)  We generated three independent GCN2-KO clones from the parental HeLa-OA1 cell line, by using three different guide RNAs, two against exon 1 (clones 183#11 and 185#5), and one against exon 6 (clone 239#1) of the GCN2 gene  Genomic characterization confirmed the presence of mutations on both alleles of exon 1 of the GCN2 gene in clone 183#11, and on both alleles of exon 6 in clone 239#1; by contrast, clone 185#5 showed multiple alleles in exon 1, consistent with the presence of two cell populations, and was not characterized further at the genomic level (S6 Fig)  None of these clones express GCN2 at the protein level, as shown by immunoblotting (Fig 6A)  To test the GCN2-KO cells for their ability to respond to EAA starvation, parental HeLa-OA1 cells and the three GCN2-KO clones were cultured in media deprived of Met/Cys or Thr (corresponding to the most effective treatments in this cell line; see Fig 3A) for 24–48 h and transgene expression was assessed by RT-qPCR  We found that the reactivation of the OA1 transgene is neither abolished, nor reduced by KO of GCN2, thus excluding that this kinase is necessary for the response to EAA starvation in HeLa-OA1 cells (Fig 6B and 6C) 
Fig 6  GCN2 knockout does not interfere with transgene reactivation in HeLa cells 
(A) Immunoblotting of protein extracts from the HeLa-OA1 parental cell line and GCN2-KO clones 183#11, 185#5 and 239#1, immunodecorated with anti-GCN2 antibody  Arrow, GCN2 specific band  Ponceau staining was used as loading control  (B, C) Relative transgene (OA1) mRNA abundance in HeLa-OA1 cells and GCN2-KO clones, cultured in Met/Cys (B) or Thr (C) deprived medium for 24 h or 48 h, respectively, compared to full medium  Mean ± SD of 3 technical replicates from 1 experiment  Data are expressed as fold change vs  control (full medium = 1)  Since independent clones may display variable reactivation responses (e g  due to different levels of transgene expression in basal conditions), the results are not shown as means of the three clones, but as separate replicates 
Similarly, we generated GCN2-KO clones from the parental HepG2-OA1 cell line by the same strategy  By using a guide RNA against exon 1 of the GCN2 gene, we obtained three independent GCN2-KO clones, namely E23, F22 and F27  Genomic characterization confirmed the presence of mutations on both alleles of exon 1 of the GCN2 gene in clone F27 (S7 Fig) and all three clones showed a very low amount—if any—of residual GCN2 protein, compared to the original HepG2-OA1 cell line (Fig 7A)  To assess the ability of GCN2-KO cells to reactivate the transgene upon starvation, we cultured parental HepG2-OA1 cells and the three GCN2-KO clones in media deprived of Met/Cys or His (corresponding to the most effective treatments in this cell line; see Fig 3B) for 24 h, and evaluated the transgene expression by RT-qPCR  As shown in Fig 7B and 7C, we found that the reactivation of the OA1 transgene is neither abolished, nor reduced by KO of GCN2, as in HeLa cells  To further confirm this result, we knocked-down GCN2 by RNA interference (RNAi), and incubated the cells with or without L-Histidinol for 6 h  As shown in Fig 8, treatment of HepG2-OA1 cells with L-Histidinol results in efficient transgene reactivation, even upon significant GCN2 downregulation, both at the mRNA and protein levels  Taken together, these data strongly support the conclusion that GCN2 is not necessary for transgene reactivation in response to EAA starvation, either in HeLa or in HepG2 cells 
Fig 7  GCN2 knockout does not interfere with transgene reactivation in HepG2 cells 
(A) Immunoblotting of protein extracts from the HepG2-OA1 parental cell line and GCN2-KO clones 185#27, E23, F22, F27, immunodecorated with anti-GCN2 antibody  Clone 185#27 results from the first round of selection, and was used to generate clones E23, F22, F27  Arrow, GCN2 specific band  For GCN2 protein quantification, Ponceau staining was used as loading control and data are expressed as fold change vs  parental cell line (= 1)  (B, C) Relative transgene (OA1) mRNA abundance in HepG2-OA1 cells and GCN2-KO clones, cultured in Met/Cys (B) or His (C) deprived medium for 24 h, compared to full medium  Mean ± SD of 3 technical replicates from 1 experiment 



ground_truth: No, it is not necessary.