Repository: pyscript/pyscript

Issue #878: display() sends the output to the wrong tag
Created: 2022-10-22T09:42:06Z
<rant>
During the infinite discussion in #769 we decided to use a "dynamic" implicit target for display(), which as far as I know can be only implemented with a global variable.
I am still convinced that this approach is very fragile and it will create tons of problems and nasty and subtle bugs.
I predict this is the first of many 🔮
We are still in time to change our minds.
</rant>
I found this problem during an unrelated refactoring. I think I know what happens but I don't want to solve it now, so I'm writing an issue to make sure we don't forget.
Consider this test:
        self.pyscript_run(
            """
            <py-script>
                display('hello 1')
            </py-script>
            <p>hello 2</p>
            <py-script>
                display('hello 3')
            </py-script>
        """
        )
It displays hello 2, hello 1, hello 3:

I think the bug was introduce by PR #749 but the roots of the bug are much deeper than this and shows some criticality on our code base.
To start with, the current test that we have is suboptimal:

  
    
      pyscript/pyscriptjs/tests/integration/test_02_output.py
    
    
        Lines 19 to 38
      in
      aa85f5f
    
  
  
    

        
          
               def test_consecutive_display(self): 
        

        
          
                   self.pyscript_run( 
        

        
          
                       """ 
        

        
          
                       <py-script> 
        

        
          
                           display('hello 1') 
        

        
          
                       </py-script> 
        

        
          
                       <py-script> 
        

        
          
                           display('hello 2') 
        

        
          
                       </py-script> 
        

        
          
                   """ 
        

        
          
                   ) 
        

        
          
                   # need to improve this to get the first/second input 
        

        
          
                   # instead of just searching for it in the page 
        

        
          
                   inner_html = self.page.content() 
        

        
          
                   first_pattern = r'<div id="py-.*?-2">hello 1</div>' 
        

        
          
                   assert re.search(first_pattern, inner_html) 
        

        
          
                   second_pattern = r'<div id="py-.*?-3">hello 2</div>' 
        

        
          
                   assert re.search(second_pattern, inner_html) 
        

        
          
            
        

        
          
                   assert first_pattern is not second_pattern 
        
    
  


it seems to test for the right thing, but because of the mess that we do when generating outputs, the test passes "by chance". The problem is that to display outputs we create <div>s inside <div>s inside <div>s, each with a random ID, which makes everything very hard to test. We should improve testability of such a simple thing. This will hopefully be solved as part of the PR I'm working on.
The other problem is much deeper. I think that the root cause is here:

  
    
      pyscript/pyscriptjs/src/components/base.ts
    
    
        Lines 119 to 124
      in
      aa85f5f
    
  
  
    

        
          
           try { 
        

        
          
               <string>await runtime.run(`set_current_display_target(target_id="${this.id}")`); 
        

        
          
               <string>await runtime.run(source); 
        

        
          
           } finally { 
        

        
          
               <string>await runtime.run(`set_current_display_target(target_id=None)`); 
        

        
          
           } 
        
    
  


The thing to note is that Runtime.run is async, so the three calls are not executed in strict order. I think that what happens is roughly the following:

main.ts starts to execute the first py-script. We run set_current_display_target(FIRST_TAG), but since it's async we immediately interrupt the execution
main.ts starts to execute the second py-scritp. We run set_current_display_target(SECOND_TAG).
at some point in the future we execute the rest of the run(), but now the "current display target" is always SECOND_TAG.

The proper fix is to ensure that the three calls are done atomically, but it's very hard to to it reliably.
A quick&dirty fix is probably to concatenate the three strings into one big string, but it's a very slipper slope. It might work right now because we execute everything in the global namespace (and we shouldn't) and it will break as soon as we stop doing it (e.g. to implement custom namespaces).
Another "maybe fix" is to use the famous ContextVar, but AFAIK it's a python-only thing. I don't really know how it interacts with the JS event loop and I think we should be very cautious before using it without fully understanding how it works in pyodide.
Another possible fix is to use synchronous calls to run: this probably solves the problem, but it breaks all py-script tags which uses await at top level. FWIW, this is one more data point which indicates that maybe they should not be allowed.
Note that currently we don't have any way to run pyodide code synchronously, which should probably kill Runtime.run and introduce Runtime.runSync and Runtime.runAsync.

Duplicate issue #932: target_id is overwritten when id is specified by the user
Created: 2022-11-09T18:42:44Z
Checklist

 I added a descriptive title
 I searched for other issues and couldn't find a solution or duplication
 I already searched in Google and didn't find any good information or help

What happened?
Seems like the async nature of our runtime is not functioning the way we expect to. (This is a guess, I describe the bug after giving some context).
I found this bug while trying to remove the dependency on ids with this PR.
I don't intend to remove all of our id generation but just prune it a little bit, removing its overuse.
Here's an example that clarifies what I mean by overuse, consider the following code:
<py-script id="first">
    display('hello 1')
</py-script>


This is the DOM it generates:

<py-script id='first'> 
  <div id='first-1'>hello 1</div>
</py-script> 

My thinking is that having a top id for the py-script tag + a generated id for the new div is too much. We should have only one for the py-script tag and none for its children. IIRC that's also @tedpatrick's take on id usage . (At least for now that's the plan, we can also think about completely extinguishing them, but that's a different discussion).
Okay. So that was the context, let's dive in to the problem I'm seeing. If you have the following code:
<py-script id="first">
    display('hello 1')
</py-script>
    <p>hello in between 1 and 2</p>
<py-script id="second">
    display('hello 2')
</py-script>
<py-script id="third">
    display('hello 3')
</py-script>

You will get the following DOM tree:
<py-script id="first"></py-script>
<p>hello in between 1 and 2</p>
<py-script id="second"></py-script>
<py-script id="third">
  <div id="third-1">hello 1</div>
  <div id="third-2">hello 2</div>
  <div id="third-3">hello 3</div>
</py-script>

And this is just wrong.
So, again, I found out about this bug on my work about id removal. In the case of my WIP PR something like this causes the page to just overwrite all the printed context since the id it focuses is always third, to the opposite of third-1, third-2, third-3.
You see, the problem is not necessarily the way we select the output tag, the problem is that once we reach Element.write() in the pyscript.py file, our dynamic id captured in the following line 
  
    
      pyscript/pyscriptjs/src/pyexec.ts
    
    
         Line 9
      in
      06a5a54
    
  
  
    

        
          
           const set_current_display_target = runtime.globals.get('set_current_display_target'); 
        
    
  

 of pyexec.ts was overwritten.
I'm not entirely sure of what's happening here, but my guess is that it's related to the nature of async?

  
    
      pyscript/pyscriptjs/src/pyexec.ts
    
    
         Line 14
      in
      06a5a54
    
  
  
    

        
          
           return await runtime.run(pysrc); 
        
    
  


Very happy to hear your thoughts on this because I'm a bit lost.
Cheers folks!
What browsers are you seeing the problem on? (if applicable)
No response
Console info
No response
Additional Context
No response

Closing comment (2022-11-09T19:42:39Z): I think this is a duplicate of #878