Repository: microsoft/botbuilder-python

Issue:
[PORT] Fix bot state so that the context service key is used consistently to access the cached state
> Port this change from botbuilder-dotnet/master branch:
https://github.com/microsoft/botbuilder-dotnet/pull/3727

Fixes #3643

A bot state class's "context service key" is used to access the cached bot state in the turn state collection. Because the context service key is usually the name of the type (e.g. "UserState") there were some cases where the SDK was just using the type name instead of the actual context service key. This fails when a bot state class has a context service key that's different from the name of the type, as my new tests show.

~~In the case of a bot state memory scope, I also fixed the code to not rely on reflection to access the cached bot state. `BotStateMemoryScope<T>.GetMemory` now accesses the bot state instance in the same way that `BotStateMemoryScope<T>.LoadAsync` and `BotStateMemoryScope<T>.SaveChangesAsync` do, and once it has the bot state instance it can simply call `BotState.Get`. Because the classes that call `GetMemory` are expecting a dictionary and not a JObject, I modified `BotState.Get` to return an `IDictionary<string, object>`. I believe this is an improvement because that's the form that the cached bot state is already in and there's no need to convert it to a JObject. While it may seem like a dangerous change, the only place that `BotState.Get` is used is in one InspectionMiddleware method, and if any customers are using it and expecting a JObject then the compiler will just tell them to convert it to a JObject on their end, so there won't be any surprising behavior changes. I'm confident that the code is better this way and I can't imagine this breaking anything because the tests are all passing.~~

I have discovered that the problems I was seeing in `BotStateMemoryScope<T>` were not about `GetMemory` returning a JObject instead of a dictionary, they were about `BotState.Get` returning a copy of the of the state, which means any modifications wouldn't persist. I have also discovered that my "best option" to fix `BotStateMemoryScope<T>` by having `Get` return the original dictionary instead of a JObject will not work because it violates the contract. I see four other options:

1. Create a new public `Get` method (with a different name) to return the actual dictionary instead of a copy
2. Create an internal `Get` method and have the Bot Builder assembly share its members with the dialogs library
3. Create a private `Get` method and have `BotStateMemoryScope<T>.GetMemory` access it using reflection
4. Don't create any new methods and have `BotStateMemoryScope<T>.GetMemory` use reflection twice

I went with option 4 because it's the most similar to the way things are currently set up. The reason we need to use reflection twice is because the `CachedBotState` class is private and the context service key we need to access it is also private.

EDIT: I've thought of a fifth option. Since bot state is meant to be accessed using bot state property accessors, we could try to rework the adaptive dialogs library to access bot state using bot state property accessors. This would probably represent significant work/changes so I would not attempt it without some kind of go-ahead from the SDK team.


# Changed projects
* Microsoft.Bot.Builder.Dialogs
* Microsoft.Bot.Builder
* Microsoft.Bot.Builder.Dialogs.Tests
* Microsoft.Bot.Builder.Tests






Discussion:

Resolving patch:
diff --git a/libraries/botbuilder-core/botbuilder/core/bot_state.py b/libraries/botbuilder-core/botbuilder/core/bot_state.py
--- a/libraries/botbuilder-core/botbuilder/core/bot_state.py
+++ b/libraries/botbuilder-core/botbuilder/core/bot_state.py
@@ -6,6 +6,7 @@
 from typing import Callable, Dict, Union
 from jsonpickle.pickler import Pickler
 from botbuilder.core.state_property_accessor import StatePropertyAccessor
+from .bot_assert import BotAssert
 from .turn_context import TurnContext
 from .storage import Storage
 from .property_manager import PropertyManager
@@ -61,6 +62,18 @@ def __init__(self, storage: Storage, context_service_key: str):
         self._storage = storage
         self._context_service_key = context_service_key
 
+    def get_cached_state(self, turn_context: TurnContext):
+        """
+        Gets the cached bot state instance that wraps the raw cached data for this "BotState"
+        from the turn context.
+
+        :param turn_context: The context object for this turn.
+        :type turn_context: :class:`TurnContext`
+        :return: The cached bot state instance.
+        """
+        BotAssert.context_not_none(turn_context)
+        return turn_context.turn_state.get(self._context_service_key)
+
     def create_property(self, name: str) -> StatePropertyAccessor:
         """
         Creates a property definition and registers it with this :class:`BotState`.
@@ -75,7 +88,8 @@ def create_property(self, name: str) -> StatePropertyAccessor:
         return BotStatePropertyAccessor(self, name)
 
     def get(self, turn_context: TurnContext) -> Dict[str, object]:
-        cached = turn_context.turn_state.get(self._context_service_key)
+        BotAssert.context_not_none(turn_context)
+        cached = self.get_cached_state(turn_context)
 
         return getattr(cached, "state", None)
 
@@ -88,10 +102,9 @@ async def load(self, turn_context: TurnContext, force: bool = False) -> None:
         :param force: Optional, true to bypass the cache
         :type force: bool
         """
-        if turn_context is None:
-            raise TypeError("BotState.load(): turn_context cannot be None.")
+        BotAssert.context_not_none(turn_context)
 
-        cached_state = turn_context.turn_state.get(self._context_service_key)
+        cached_state = self.get_cached_state(turn_context)
         storage_key = self.get_storage_key(turn_context)
 
         if force or not cached_state or not cached_state.state:
@@ -111,10 +124,9 @@ async def save_changes(
         :param force: Optional, true to save state to storage whether or not there are changes
         :type force: bool
         """
-        if turn_context is None:
-            raise TypeError("BotState.save_changes(): turn_context cannot be None.")
+        BotAssert.context_not_none(turn_context)
 
-        cached_state = turn_context.turn_state.get(self._context_service_key)
+        cached_state = self.get_cached_state(turn_context)
 
         if force or (cached_state is not None and cached_state.is_changed):
             storage_key = self.get_storage_key(turn_context)
@@ -134,8 +146,7 @@ async def clear_state(self, turn_context: TurnContext):
         .. remarks::
             This function must be called in order for the cleared state to be persisted to the underlying store.
         """
-        if turn_context is None:
-            raise TypeError("BotState.clear_state(): turn_context cannot be None.")
+        BotAssert.context_not_none(turn_context)
 
         #  Explicitly setting the hash will mean IsChanged is always true. And that will force a Save.
         cache_value = CachedBotState()
@@ -151,8 +162,7 @@ async def delete(self, turn_context: TurnContext) -> None:
 
         :return: None
         """
-        if turn_context is None:
-            raise TypeError("BotState.delete(): turn_context cannot be None.")
+        BotAssert.context_not_none(turn_context)
 
         turn_context.turn_state.pop(self._context_service_key)
 
@@ -174,15 +184,12 @@ async def get_property_value(self, turn_context: TurnContext, property_name: str
 
         :return: The value of the property
         """
-        if turn_context is None:
-            raise TypeError(
-                "BotState.get_property_value(): turn_context cannot be None."
-            )
+        BotAssert.context_not_none(turn_context)
         if not property_name:
             raise TypeError(
                 "BotState.get_property_value(): property_name cannot be None."
             )
-        cached_state = turn_context.turn_state.get(self._context_service_key)
+        cached_state = self.get_cached_state(turn_context)
 
         # if there is no value, this will throw, to signal to IPropertyAccesor that a default value should be computed
         # This allows this to work with value types
@@ -201,11 +208,10 @@ async def delete_property_value(
 
         :return: None
         """
-        if turn_context is None:
-            raise TypeError("BotState.delete_property(): turn_context cannot be None.")
+        BotAssert.context_not_none(turn_context)
         if not property_name:
             raise TypeError("BotState.delete_property(): property_name cannot be None.")
-        cached_state = turn_context.turn_state.get(self._context_service_key)
+        cached_state = self.get_cached_state(turn_context)
         del cached_state.state[property_name]
 
     async def set_property_value(
@@ -223,11 +229,10 @@ async def set_property_value(
 
         :return: None
         """
-        if turn_context is None:
-            raise TypeError("BotState.delete_property(): turn_context cannot be None.")
+        BotAssert.context_not_none(turn_context)
         if not property_name:
             raise TypeError("BotState.delete_property(): property_name cannot be None.")
-        cached_state = turn_context.turn_state.get(self._context_service_key)
+        cached_state = self.get_cached_state(turn_context)
         cached_state.state[property_name] = value
 
 


Test patch:
diff --git a/libraries/botbuilder-core/tests/test_bot_state.py b/libraries/botbuilder-core/tests/test_bot_state.py
--- a/libraries/botbuilder-core/tests/test_bot_state.py
+++ b/libraries/botbuilder-core/tests/test_bot_state.py
@@ -473,13 +473,32 @@ async def test_bot_state_get(self):
 
         storage = MemoryStorage({})
 
-        conversation_state = ConversationState(storage)
+        test_bot_state = BotStateForTest(storage)
         (
-            await conversation_state.create_property("test-name").get(
+            await test_bot_state.create_property("test-name").get(
                 turn_context, lambda: TestPocoState()
             )
         ).value = "test-value"
 
-        result = conversation_state.get(turn_context)
+        result = test_bot_state.get(turn_context)
 
         assert result["test-name"].value == "test-value"
+
+    async def test_bot_state_get_cached_state(self):
+        # pylint: disable=unnecessary-lambda
+        turn_context = TestUtilities.create_empty_context()
+        turn_context.activity.conversation = ConversationAccount(id="1234")
+
+        storage = MemoryStorage({})
+
+        test_bot_state = BotStateForTest(storage)
+        (
+            await test_bot_state.create_property("test-name").get(
+                turn_context, lambda: TestPocoState()
+            )
+        ).value = "test-value"
+
+        result = test_bot_state.get_cached_state(turn_context)
+
+        assert result is not None
+        assert result == test_bot_state.get_cached_state(turn_context)