Repository: revbayes/revbayes
Pull request #719: Fix deterministic index
Author: login: bredelings
Created: 2025-04-12T00:12:40Z | Merged: 2025-04-14T17:24:54Z

In revbayes script you can write:
```
i ~ dnCategorical([0.1,0.2,0.3,0.4])
for(k in 1:10)
{
  x[k] ~ dnNormal(0,1)
}
i.setValue(1)
j := i*2
y := x[i]
z := x[j]
w := x[2*i]
```

Currently, when revbayes sees a node like `x[i]` it creates a node name like `x[1]` based on the initial value of `i`.  If that node already exists, then revbayes will just use that instead of creating a deterministic node for `x[i]`.  This is OK if `i` is constant, but produces unexpected behavior when `i` is not a constant.

This PR fixes this by only short-circuiting to an existing node like `x[1]` when the DagNode for the index is _not_ deterministic or stochastic.  Additionally, the PR tries to compute a more sensible name for the node `x[i]` that does not just use the initial value of `i` to compute the name.

---
I moved a companion fix to #721 



Diff:
diff --git a/src/revlanguage/parser/SyntaxIndexOperation.cpp b/src/revlanguage/parser/SyntaxIndexOperation.cpp
index 4c48a4bd9b..1f923d2983 100644
--- a/src/revlanguage/parser/SyntaxIndexOperation.cpp
+++ b/src/revlanguage/parser/SyntaxIndexOperation.cpp
@@ -230,25 +230,31 @@ RevPtr<RevVariable> SyntaxIndexOperation::evaluateLHSContent( const std::shared_
  */
 RevPtr<RevVariable> SyntaxIndexOperation::evaluateContent( const std::shared_ptr<Environment>& env, bool dynamic)
 {
-
     RevPtr<RevVariable> indexVar     = index->evaluateContent(env,dynamic);
     RevPtr<RevVariable> theParentVar = base_variable->evaluateContent( env );
-    std::string identifier = theParentVar->getName() + "[" + indexVar->getRevObject().toString() + "]";
-
     RevPtr<RevVariable> the_var = NULL;
 
+    bool constant_index = true;
+    if (indexVar->getRevObject().hasDagNode())
+        constant_index = indexVar->getRevObject().getDagNode()->isConstant();
+
+    string index_name = "_"; // Choose a character that doesn't break model.graph(filename).
+    if (constant_index)
+        index_name = indexVar->getRevObject().toString();
+    else if (not indexVar->getName().empty())
+        index_name = indexVar->getName();
+
+    std::string identifier = theParentVar->getName() + "[" + index_name + "]";
+
     // test whether we want to directly assess the variable or if we want to assess subelement of this container
-    // if this variable already exists in the workspace
-    if ( env->existsVariable( identifier ) )
+    // if this variable already exists in the workspace and the index cannot change
+    if ( env->existsVariable( identifier ) and constant_index)
     {
         // get the from the workspace
         the_var = env->getVariable( identifier );
-
     }
     else
     {
-
-
         try
         {
             // convert the value into a member object
diff --git a/tests/test_deterministic_index/output_expected/test.errout b/tests/test_deterministic_index/output_expected/test.errout
new file mode 100644
index 0000000000..de2f010042
--- /dev/null
+++ b/tests/test_deterministic_index/output_expected/test.errout
@@ -0,0 +1,16 @@
+
+Setting i to 1:
+
+i = 1
+j = 2
+x[i] = 1.59852
+x[j] = 0.32888
+x[2*i] = 0.32888
+
+Setting i to 3:
+
+i = 3
+j = 6
+x[i] = 0.142168
+x[j] = -1.0983
+x[2*i] = -1.0983
diff --git a/tests/test_deterministic_index/scripts/test.Rev b/tests/test_deterministic_index/scripts/test.Rev
new file mode 100644
index 0000000000..d8a8197a70
--- /dev/null
+++ b/tests/test_deterministic_index/scripts/test.Rev
@@ -0,0 +1,34 @@
+seed(0)
+i ~ dnCategorical(v(0.5, 0.2, 0.3))
+for(k in 1:10)
+{
+  x[k] ~ dnNormal(0,1)
+}
+
+j := i*2
+y := x[i]
+z := x[j]
+w := x[2*i]
+
+model(i).graph("output/model.dot")
+
+print("\nSetting i to 1:\n")
+i.setValue(1)
+
+print("i = " + i)
+print("j = " + j)
+print("x[i] = " + y)
+print("x[j] = " + z)
+print("x[2*i] = " + w)
+
+
+i.setValue(3)
+print("\nSetting i to 3:\n")
+
+print("i = " +i)
+print("j = " + j)
+print("x[i] = " + y)
+print("x[j] = " + z)
+print("x[2*i] = " + w)
+
+q()
\ No newline at end of file


Reviews:
user: login: bredelings
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-12T01:13:41Z
user: login: bredelings
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-12T01:14:03Z
user: login: davidcerny
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T01:51:16Z
user: login: bredelings
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T15:27:43Z

Review comments:
diff_hunk: @@ -425,17 +425,16 @@ double ArgumentRule::isArgumentValid( Argument &arg, bool once) const
     
     // we need to store and check all arg types
     std::vector<double> penalties;
-    for ( std::vector<TypeSpec>::const_iterator it = argTypeSpecs.begin(); it != argTypeSpecs.end(); ++it )
+    for (auto& req_arg_type_spec: argTypeSpecs)
     {
-        const TypeSpec& req_arg_type_spec = *it;
         if ( the_var->getRevObject().isType( req_arg_type_spec ) )
         {
             return 0.0;
         }
             
         double penalty = -1;
         // make sure that we only perform type casting when the variable will not be part of a model graph
-        if ( once == true || the_var->getRevObject().isConstant() == true )
+        if ( the_var->getRevObject().isConstant() == true )
path: src/revlanguage/functions/argumentrules/ArgumentRule.cpp
user: login: bredelings
body: This is the line that changes the type-conversion rules.
created_at: 2025-04-12T01:13:41Z
author_association: CONTRIBUTOR
diff_hunk: @@ -230,25 +230,31 @@ RevPtr<RevVariable> SyntaxIndexOperation::evaluateLHSContent( const std::shared_
  */
 RevPtr<RevVariable> SyntaxIndexOperation::evaluateContent( const std::shared_ptr<Environment>& env, bool dynamic)
 {
-
     RevPtr<RevVariable> indexVar     = index->evaluateContent(env,dynamic);
     RevPtr<RevVariable> theParentVar = base_variable->evaluateContent( env );
-    std::string identifier = theParentVar->getName() + "[" + indexVar->getRevObject().toString() + "]";
-
     RevPtr<RevVariable> the_var = NULL;
 
+    bool constant_index = true;
+    if (indexVar->getRevObject().hasDagNode())
+        constant_index = indexVar->getRevObject().getDagNode()->isConstant();
+
+    string index_name = "_"; // Choose a character that doesn't break model.graph(filename).
+    if (constant_index)
+        index_name = indexVar->getRevObject().toString();
+    else if (not indexVar->getName().empty())
+        index_name = indexVar->getName();
+
+    std::string identifier = theParentVar->getName() + "[" + index_name + "]";
+
     // test whether we want to directly assess the variable or if we want to assess subelement of this container
-    // if this variable already exists in the workspace
-    if ( env->existsVariable( identifier ) )
+    // if this variable already exists in the workspace and the index cannot change
+    if ( env->existsVariable( identifier ) and constant_index)
path: src/revlanguage/parser/SyntaxIndexOperation.cpp
user: login: bredelings
body: This is the line that changes how `x[i]` is handled.
created_at: 2025-04-12T01:14:03Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,34 @@
+seed(0)
+i ~ dnCategorical(v(0.5, 0.2, 0.3))
+for(k in 1:10)
+{
+  x[k] ~ dnNormal(0,1)
+}
+
+j := i*2
+y := x[i]
+z := x[j]
+w := x[2*i]
+
+model(i).graph("output/model.dot")
path: tests/test_deterministic_index/scripts/test.Rev
user: login: davidcerny
body: Should `model.dot` be included in the expected output?
created_at: 2025-04-14T01:05:48Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,34 @@
+seed(0)
+i ~ dnCategorical(v(0.5, 0.2, 0.3))
+for(k in 1:10)
+{
+  x[k] ~ dnNormal(0,1)
+}
+
+j := i*2
+y := x[i]
+z := x[j]
+w := x[2*i]
+
+model(i).graph("output/model.dot")
path: tests/test_deterministic_index/scripts/test.Rev
user: login: bredelings
body: I originally wanted it to be.  That is one way to ensure that the dependencies between `x[i]` and `i` are tracked.

But it turns out that names for the graph nodes are generated from pointers if the node is unnamed.  So the `model.dot` always changes even if the seed is constant.

In the future I would like to fix this, and then we can put `model.dot` in the expected output.  I will file an issue...
created_at: 2025-04-14T15:27:43Z
author_association: CONTRIBUTOR

PR comments:
user: login: davidcerny
created_at: 2025-04-14T01:50:55Z
body: I've tested this on a script (attached [here](https://github.com/user-attachments/files/19728324/SHDM_rjMCMC.zip) along with the corresponding data) from an ongoing collaboration with @wrightaprilm, in which we use rjMCMC sampling to average equilibrium state frequencies over a mixture of different Dirichlet distributions. Following the [corresponding tutorial](https://revbayes.github.io/tutorials/model_selection_bayes_factors/bf_subst_model_rj.html), the script implements the model-averaging step as follows:

```r
model_indicator ~ dnCategorical( simplex( rep(1, alpha_vals.size()) ) )
moves.append( mvRandomGeometricWalk(model_indicator, weight=10.0) )

alpha_dir := alpha_vals[model_indicator]
```

On the current tip of `development`, this gives me the same error that was noted in discussion #608, that is:

```
   Error:	Cannot find node with name 'model_indicator' in the model but received a move working on it.
```

The only way to get around this problem is to use the alternative rjMCMC implementation suggested in the tutorial:

```r
alpha_dir ~ dnMixture( values=alpha_vals, probabilities=simplex( rep(1, alpha_vals.size()) ) )
model_indicator := alpha_dir.getAllocationIndex()

moves.append( mvGibbsMixtureAllocation(alpha_dir, weight=2.0) )
```

With this PR, the script seemingly worked just fine even with the former solution based on `dnCategorical()` rather than `dnMixture()`. However, when I ran it with `--setOption debugMCMC=1 --setOption logMCMC=3`, I got an issue similar to the one noted in #664:

```
   Error:	Index out of bounds: The vector of size 6 does not have an element for index 8.
```

I then merged the `fix-out-of-range-proposal` branch (see PR #709) into this one and re-ran the analysis. That still resulted in many "index out of bounds" errors:

```
    PROPOSED: model_indicator:  -inf
    PROPOSED: pi[1][2]:  Exception: Index out of bounds: The vector of size 6 does not have an element for index 0.
    PROPOSED: pi[1][4]:  Exception: Index out of bounds: The vector of size 6 does not have an element for index 0.
    PROPOSED: pi[1][3]:  Exception: Index out of bounds: The vector of size 6 does not have an element for index 0.
    PROPOSED: pi[1][5]:  Exception: Index out of bounds: The vector of size 6 does not have an element for index 0.
    ...
```

but now the MCMC was able to recover from those and seemed to run just fine.

I can't entirely rule out the possibility that some silent failures might still be taking place. In particular, while a draw from `dnCategorical()` is of type `Natural`, the script then allows it to be moved by `mvRandomGeometricWalk`, which expects an argument of type `Integer`. So perhaps the changes made in PR #721 would be needed as well to ensure that everything really works as it should.

In any case, it's clear to me from the above that between the three of them, this PR as well as #709 and #721 represent a huge step toward fixing one of the basic mechanisms we have for conducting rjMCMC analyses in RevBayes. I'm in favor of merging them sooner rather than later.
author_association: CONTRIBUTOR
user: login: thauffe
created_at: 2025-04-14T14:33:45Z
body: Thanks for your feedback David!

Do you see different values for the model_indicator in your log file? I run it only for a few generations but the value stayed constant.





author_association: NONE
user: login: bredelings
created_at: 2025-04-14T15:59:00Z
body: Interesting!  Thanks for exploring this!
author_association: CONTRIBUTOR
user: login: davidcerny
created_at: 2025-04-14T18:31:59Z
body: @thauffe The analysis is very slow and the parameter doesn't seem to mix particularly well, but there is _some_ movement, so it doesn't look like its estimation is fundamentally broken:

<img width="914" alt="Screenshot 2025-04-14 at 13 27 41" src="https://github.com/user-attachments/assets/725a14ee-53b3-493b-a4da-26af7ac91eb3" />

author_association: CONTRIBUTOR