Repository: sonic-net/sonic-gnmi
Pull request #375: Fix poll mode for querying non existing table or key for APPL_DB
Author: login: zbud-msft
Labels: name: Cherry Pick Conflict_202405
name: Request for 202405 Branch
name: Request for 202411 Branch
name: Approved for 202411 Branch
name: Included in 202411 Branch
name: Request for msft-202405 Branch
Created: 2025-04-15T07:51:15Z | Merged: 2025-04-18T17:40:55Z

<!--
     Please make sure you've read and understood our contributing guidelines:
     https://github.com/Azure/SONiC/blob/gh-pages/CONTRIBUTING.md

     ** Make sure all your commits include a signature generated with `git commit -s` **

     If this is a bug fix, make sure your description includes "fixes #xxxx", or
     "closes #xxxx" or "resolves #xxxx"

     Please provide the following information:
-->

#### Why I did it

MS: 32418202

SONiC telemetry does not behave according to gnmi specifications

| **POLL MODE**                   | **Target doesn't exist**                                      | **Table doesn't exist**                                                                                                              | **Key doesn't exist**                                                                                                      | **Table gets deleted**                        | **Key gets deleted**                                                                               | **Field added**                | **Field gets changed**            |
|---------------------------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------|---------------------------------|-------------------------------------|
| **Official gNMI Spec**          | RPC_NOT_FOUND, entire RPC fails                               | Omit Notification from RPC response, rpc should be open                                                                              | Omit notification from RPC response, rpc should be open                                                                     | Delete notifcation includes path               | Delete notification includes path                                                                                             | Normal update response         | Normal update response             |
| **SONiC**                       | RPC_NOT_FOUND, entire RPC fails                               | RPC stays open and updates work properly. We do send empty notifications which isn't technically part of the official gnmi protocol. | Server fails internally with redis Hget, client is left to hang with no sync response; RPC is left in a bad state with no updates even from good paths. | Empty update notifications. No delete proto    | Update notification with remaining keys or just empty update notification. No delete proto                                     | Normal update response         | Normal Update response             |


https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-specification.md#3513-the-subscription-message

As per gnmi specification regarding missing data, "There is no requirement that the path specified in the message must exist within the current data tree on the server. While the path within the subscription SHOULD be a valid path within the set of schema modules that the target supports, subscribing to any syntactically valid path within such modules MUST be allowed. In the case that a particular path does not (yet) exist, the target MUST NOT close the RPC and instead should continue to monitor for the existence of the path, and transmit telemetry updates should it exist in the future."

As per gnmi specification regarding deleted keys/paths, "Where a node within the subscribed paths has been removed, the delete field of the Notification message MUST have the path of the node that has been removed appended to it."

#### How I did it

If there is missing data, we will not return any failures, but instead we will send sync responses only until data is on the device for polling APPL_DB

If data is deleted, we will send a delete notification to tell client that path they are querying is deleted and after send sync responses for polling APPL_DB.

#### How to verify it

UT

#### Which release branch to backport (provide reason below if selected)

<!--
- Note we only backport fixes to a release branch, *not* features!
- Please also provide a reason for the backporting below.
- e.g.
- [x] 202006
-->

- [ ] 201811
- [ ] 201911
- [ ] 202006
- [ ] 202012
- [ ] 202106
- [ ] 202111

#### Description for the changelog
<!--
Write a short (one line) summary that describes the changes in this
pull request for inclusion in the changelog:
-->

#### Link to config_db schema for YANG module changes
<!--
Provide a link to config_db schema for the table for which YANG model
is defined
Link should point to correct section on https://github.com/Azure/SONiC/wiki/Configuration.
-->

#### A picture of a cute animal (not mandatory but encouraged)




Diff:
diff --git a/gnmi_server/client_subscribe.go b/gnmi_server/client_subscribe.go
index 72aed56c1..77eeab722 100644
--- a/gnmi_server/client_subscribe.go
+++ b/gnmi_server/client_subscribe.go
@@ -2,10 +2,6 @@ package gnmi
 
 import (
 	"fmt"
-	"io"
-	"net"
-	"sync"
-
 	"github.com/Workiva/go-datastructures/queue"
 	log "github.com/golang/glog"
 	gnmipb "github.com/openconfig/gnmi/proto/gnmi"
@@ -13,6 +9,10 @@ import (
 	"google.golang.org/grpc"
 	"google.golang.org/grpc/codes"
 	"google.golang.org/grpc/status"
+	"io"
+	"net"
+	"strings"
+	"sync"
 )
 
 // Client contains information about a subscribe client that has connected to the server.
@@ -195,7 +195,11 @@ func (c *Client) Run(stream gnmipb.GNMI_SubscribeServer) (err error) {
 		c.polled = make(chan struct{}, 1)
 		c.polled <- struct{}{}
 		c.w.Add(1)
-		go dc.PollRun(c.q, c.polled, &c.w, c.subscribe)
+		if target == "APPL_DB" || strings.HasPrefix(target, "APPL_DB/") {
+			go dc.AppDBPollRun(c.q, c.polled, &c.w, c.subscribe)
+		} else {
+			go dc.PollRun(c.q, c.polled, &c.w, c.subscribe)
+		}
 	case gnmipb.SubscriptionList_ONCE:
 		c.once = make(chan struct{}, 1)
 		c.once <- struct{}{}
diff --git a/gnmi_server/poll_mode_test.go b/gnmi_server/poll_mode_test.go
new file mode 100644
index 000000000..0ba501c1b
--- /dev/null
+++ b/gnmi_server/poll_mode_test.go
@@ -0,0 +1,1683 @@
+package gnmi
+
+// server_test covers gNMI get, subscribe (stream and poll) test
+// Prerequisite: redis-server should be running.
+import (
+	"crypto/tls"
+	"encoding/json"
+	"github.com/kylelemons/godebug/pretty"
+	"github.com/openconfig/gnmi/client"
+	sdcfg "github.com/sonic-net/sonic-gnmi/sonic_db_config"
+	"golang.org/x/net/context"
+	"io/ioutil"
+	"reflect"
+	"sync"
+	"testing"
+	"time"
+)
+
+func TestPollMissingTableThenTableKey(t *testing.T) {
+	// Test that 1) missing table 2)table + key should just send sync responses and rpc connection should be alive
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE",
+			poll: 3,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+			},
+		},
+		{
+			desc: "query ROUTE_TABLE:0.0.0.0/0",
+			poll: 3,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				if nn, ok := n.(client.Update); ok {
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				} else {
+					gotNoti = append(gotNoti, n)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 {
+				t.Errorf("Expected non zero length of notifications")
+			}
+
+			if diff := pretty.Compare(tt.wantNoti, gotNoti); diff != "" {
+				t.Log("\n Want: \n", tt.wantNoti)
+				t.Log("\n Got : \n", gotNoti)
+				t.Errorf("unexpected updates:\n%s", diff)
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollMissingTableAndTableKey(t *testing.T) {
+	// Test that missing table and table + key should just send sync responses and rpc connection should be alive
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE + ROUTE_TABLE:0.0.0.0/0",
+			poll: 3,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}, {"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				if nn, ok := n.(client.Update); ok {
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				} else {
+					gotNoti = append(gotNoti, n)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 {
+				t.Errorf("Expected non zero length of notifications")
+			}
+
+			if diff := pretty.Compare(tt.wantNoti, gotNoti); diff != "" {
+				t.Log("\n Want: \n", tt.wantNoti)
+				t.Log("\n Got : \n", gotNoti)
+				t.Errorf("unexpected updates:\n%s", diff)
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+
+}
+
+func TestPollMissingTableThenAdded(t *testing.T) {
+	// Test that missing table should just send sync responses and rpc connection should be alive
+	// When we add data for Table, we should receive update notifications
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				if nn, ok := n.(client.Update); ok {
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				} else {
+					gotNoti = append(gotNoti, n)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, add data
+					rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+					rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+					// Sleep just one second to allow redis data to be entered
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 {
+				t.Errorf("Expected non zero length of notifications")
+			}
+
+			if diff := pretty.Compare(tt.wantNoti, gotNoti); diff != "" {
+				t.Log("\n Want: \n", tt.wantNoti)
+				t.Log("\n Got : \n", gotNoti)
+				t.Errorf("unexpected updates:\n%s", diff)
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollMissingKeyThenAdded(t *testing.T) {
+	// Test that missing table+key should just send sync responses and rpc connection should be alive
+	// When we add data for table+key, we should receive update notifications
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				if nn, ok := n.(client.Update); ok {
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				} else {
+					gotNoti = append(gotNoti, n)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, add data
+					rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+					rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+					// Sleep just one second to allow redis data to be entered
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 {
+				t.Errorf("Expected non zero length of notifications")
+			}
+
+			if diff := pretty.Compare(tt.wantNoti, gotNoti); diff != "" {
+				t.Log("\n Want: \n", tt.wantNoti)
+				t.Log("\n Got : \n", gotNoti)
+				t.Errorf("unexpected updates:\n%s", diff)
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollMissingTableAndKeyThenAdded(t *testing.T) {
+	// Test that we get not updates from missing table and table key queried but still get sync responses
+	// After adding back, we will get both updates
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	fileName = "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE + ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}, {"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				if nn, ok := n.(client.Update); ok {
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				} else {
+					gotNoti = append(gotNoti, n)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, add data
+					rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+					rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+					rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+					rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+					// Sleep just one second to allow redis data to be entered
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 || len(gotNoti) != len(tt.wantNoti) {
+				t.Errorf("Expected non zero length of notifications or equal notifications")
+			}
+
+			// check that the connected and sync messages are identical
+			for i := 0; i < 4; i++ {
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) {
+					t.Fatalf("notification %d mismatch:\n got  %#v\n want %#v", i, gotNoti[i], tt.wantNoti[i])
+				}
+			}
+
+			// Check that both notifications are coming at every poll interval
+			for _, pair := range [][2]int{{4, 5}, {7, 8}, {10, 11}} { // these indexes are our update notifications
+				i, j := pair[0], pair[1]
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) || !reflect.DeepEqual(gotNoti[j], tt.wantNoti[j]) {
+					if !reflect.DeepEqual(gotNoti[j], tt.wantNoti[i]) && !reflect.DeepEqual(gotNoti[i], tt.wantNoti[j]) {
+						t.Fatalf("mismatch at indices %d/%d:\n got  (%#v, %#v)\n want (%#v, %#v)", i, j, gotNoti[i], gotNoti[j], tt.wantNoti[i], tt.wantNoti[j])
+					}
+				}
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+
+}
+
+func TestPollPresentTableMissingTableKey(t *testing.T) {
+	// Test that we receive update notification for table query and no data for missing key
+	// After 2 polls, we will add back the missing key data to get both data in our notifications
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	fileName = "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE + ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}, {"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				if nn, ok := n.(client.Update); ok {
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				} else {
+					gotNoti = append(gotNoti, n)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, add data
+					rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+					rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+					// Sleep just one second to allow redis data to be entered
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 || len(gotNoti) != len(tt.wantNoti) {
+				t.Errorf("Expected non zero length of notifications or equal notifications")
+			}
+
+			// check that the connected and sync messages and table update notifications are identical
+			for i := 0; i < 7; i++ {
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) {
+					t.Fatalf("notification %d mismatch:\n got  %#v\n want %#v", i, gotNoti[i], tt.wantNoti[i])
+				}
+			}
+
+			// Check that both notifications are coming at every poll interval
+			for _, pair := range [][2]int{{7, 8}, {10, 11}, {13, 14}} { // these indexes are our update notifications
+				i, j := pair[0], pair[1]
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) || !reflect.DeepEqual(gotNoti[j], tt.wantNoti[j]) {
+					if !reflect.DeepEqual(gotNoti[j], tt.wantNoti[i]) && !reflect.DeepEqual(gotNoti[i], tt.wantNoti[j]) {
+						t.Fatalf("mismatch at indices %d/%d:\n got  (%#v, %#v)\n want (%#v, %#v)", i, j, gotNoti[i], gotNoti[j], tt.wantNoti[i], tt.wantNoti[j])
+					}
+				}
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollPresentTableKeyMissingTable(t *testing.T) {
+	// Test that we receive update notification for table key query and no data for missing table
+	// After 2 polls, we will add back the missing table data to get both data in our notifications
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	fileName = "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE + ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}, {"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				if nn, ok := n.(client.Update); ok {
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				} else {
+					gotNoti = append(gotNoti, n)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, add data
+					rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+					rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+					// Sleep just one second to allow redis data to be entered
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 || len(gotNoti) != len(tt.wantNoti) {
+				t.Errorf("Expected non zero length of notifications or equal notifications")
+			}
+
+			// check that the connected and sync messages and table key update notifications are identical
+			for i := 0; i < 7; i++ {
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) {
+					t.Fatalf("notification %d mismatch:\n got  %#v\n want %#v", i, gotNoti[i], tt.wantNoti[i])
+				}
+			}
+
+			// Check that both notifications are coming at every poll interval
+			for _, pair := range [][2]int{{7, 8}, {10, 11}, {13, 14}} { // these indexes are our update notifications
+				i, j := pair[0], pair[1]
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) || !reflect.DeepEqual(gotNoti[j], tt.wantNoti[j]) {
+					if !reflect.DeepEqual(gotNoti[j], tt.wantNoti[i]) && !reflect.DeepEqual(gotNoti[i], tt.wantNoti[j]) {
+						t.Fatalf("mismatch at indices %d/%d:\n got  (%#v, %#v)\n want (%#v, %#v)", i, j, gotNoti[i], gotNoti[j], tt.wantNoti[i], tt.wantNoti[j])
+					}
+				}
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollTableDeleted(t *testing.T) {
+	// Test that we received update notifications for existing table data, then delete table, we should receive 1 delete notification
+	// After delete notification, we should only see sync responses
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Delete{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200)},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				switch nn := n.(type) {
+				case client.Connected, client.Sync:
+					gotNoti = append(gotNoti, nn)
+				case client.Delete:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				case client.Update:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				default:
+					t.Errorf("Unexpected Client Notification: %v", nn)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, del data
+					rclient.FlushDB()
+					// Sleep just one second to allow redis data to be deleted
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 {
+				t.Errorf("Expected non zero length of notifications")
+			}
+
+			if diff := pretty.Compare(tt.wantNoti, gotNoti); diff != "" {
+				t.Log("\n Want: \n", tt.wantNoti)
+				t.Log("\n Got : \n", gotNoti)
+				t.Errorf("unexpected updates:\n%s", diff)
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollTableFieldDeleted(t *testing.T) {
+	// Test that we received update notifications for existing table field data, then delete table field, we should receive 1 delete notification
+	// After delete notification, we should only see sync responses
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_LOC_CHASSIS + lldp_loc_sys_name",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_LOC_CHASSIS", "lldp_loc_sys_name"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_LOC_CHASSIS", "lldp_loc_sys_name"}, TS: time.Unix(0, 200), Val: "dummy"},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_LOC_CHASSIS", "lldp_loc_sys_name"}, TS: time.Unix(0, 200), Val: "dummy"},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_LOC_CHASSIS", "lldp_loc_sys_name"}, TS: time.Unix(0, 200), Val: "dummy"},
+				client.Sync{},
+				client.Delete{Path: []string{"APPL_DB", "LLDP_LOC_CHASSIS", "lldp_loc_sys_name"}, TS: time.Unix(0, 200)},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("LLDP_LOC_CHASSIS", "lldp_loc_sys_name", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				switch nn := n.(type) {
+				case client.Connected, client.Sync:
+					gotNoti = append(gotNoti, nn)
+				case client.Delete:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				case client.Update:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				default:
+					t.Errorf("Unexpected Client Notification: %v", nn)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, del data
+					rclient.FlushDB()
+					// Sleep just one second to allow redis data to be deleted
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 {
+				t.Errorf("Expected non zero length of notifications")
+			}
+
+			if diff := pretty.Compare(tt.wantNoti, gotNoti); diff != "" {
+				t.Log("\n Want: \n", tt.wantNoti)
+				t.Log("\n Got : \n", gotNoti)
+				t.Errorf("unexpected updates:\n%s", diff)
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollTableKeyDeleted(t *testing.T) {
+	// Test that we received update notifications for existing table key data, then delete table key, we should receive 1 delete notification
+	// After delete notification, we should only see sync responses
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Delete{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200)},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				switch nn := n.(type) {
+				case client.Connected, client.Sync:
+					gotNoti = append(gotNoti, nn)
+				case client.Delete:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				case client.Update:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				default:
+					t.Errorf("Unexpected Client Notification: %v", nn)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, del data
+					rclient.FlushDB()
+					// Sleep just one second to allow redis data to be deleted
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 {
+				t.Errorf("Expected non zero length of notifications")
+			}
+
+			if diff := pretty.Compare(tt.wantNoti, gotNoti); diff != "" {
+				t.Log("\n Want: \n", tt.wantNoti)
+				t.Log("\n Got : \n", gotNoti)
+				t.Errorf("unexpected updates:\n%s", diff)
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollTableAndTableKeyBothDeleted(t *testing.T) {
+	// Test that we received update notifications for existing data, then delete both table and table key, we should receive 2 delete notifications
+	// After delete notifications, we should only see sync responses
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	fileName = "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE + ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}, {"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Delete{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200)},
+				client.Delete{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200)},
+				client.Sync{},
+				client.Sync{},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				switch nn := n.(type) {
+				case client.Connected, client.Sync:
+					gotNoti = append(gotNoti, nn)
+				case client.Delete:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				case client.Update:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				default:
+					t.Errorf("Unexpected Client Notification: %v", nn)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, delete data
+					rclient.FlushDB()
+					// Sleep just one second to allow redis data to be deleted
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 || len(gotNoti) != len(tt.wantNoti) {
+				t.Errorf("Expected non zero length of notifications or equal notifications")
+			}
+
+			// Check that both notifications are coming at every poll interval
+			for _, pair := range [][2]int{{1, 2}, {4, 5}, {7, 8}, {10, 11}} { // these indexes are our update notifications
+				i, j := pair[0], pair[1]
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) || !reflect.DeepEqual(gotNoti[j], tt.wantNoti[j]) {
+					if !reflect.DeepEqual(gotNoti[j], tt.wantNoti[i]) && !reflect.DeepEqual(gotNoti[i], tt.wantNoti[j]) {
+						t.Fatalf("mismatch at indices %d/%d:\n got  (%#v, %#v)\n want (%#v, %#v)", i, j, gotNoti[i], gotNoti[j], tt.wantNoti[i], tt.wantNoti[j])
+					}
+				}
+			}
+
+			// check that the sync messages are identical
+			for i := 12; i < 15; i++ {
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) {
+					t.Fatalf("notification %d mismatch:\n got  %#v\n want %#v", i, gotNoti[i], tt.wantNoti[i])
+				}
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollTableAndTableKeyTableDeleted(t *testing.T) {
+	// Test that we receive update notifications for existing data, and then when we delete table we should receive delete notification
+	// After delete notification, we should see sync responses and continued update for existing data
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	fileName = "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE + ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}, {"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Delete{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200)},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				switch nn := n.(type) {
+				case client.Connected, client.Sync:
+					gotNoti = append(gotNoti, nn)
+				case client.Delete:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				case client.Update:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				default:
+					t.Errorf("Unexpected Client Notification: %v", nn)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, delete data
+					rclient.Del("LLDP_ENTRY_TABLE:eth0")
+					// Sleep just one second to allow redis data to be deleted
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 || len(gotNoti) != len(tt.wantNoti) {
+				t.Errorf("Expected non zero length of notifications or equal notifications")
+			}
+
+			// Check that both notifications are coming at every poll interval
+			for _, pair := range [][2]int{{1, 2}, {4, 5}, {7, 8}, {10, 11}} { // these indexes are our update notifications
+				i, j := pair[0], pair[1]
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) || !reflect.DeepEqual(gotNoti[j], tt.wantNoti[j]) {
+					if !reflect.DeepEqual(gotNoti[j], tt.wantNoti[i]) && !reflect.DeepEqual(gotNoti[i], tt.wantNoti[j]) {
+						t.Fatalf("mismatch at indices %d/%d:\n got  (%#v, %#v)\n want (%#v, %#v)", i, j, gotNoti[i], gotNoti[j], tt.wantNoti[i], tt.wantNoti[j])
+					}
+				}
+			}
+
+			// check that the sync messages are identical
+			for i := 13; i < 17; i++ {
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) {
+					t.Fatalf("notification %d mismatch:\n got  %#v\n want %#v", i, gotNoti[i], tt.wantNoti[i])
+				}
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
+
+func TestPollTableAndTableKeyTableKeyDeleted(t *testing.T) {
+	// Test that we receive update notifications for existing data, and then when we delete table key we should receive delete notification
+	// After delete notification, we should see sync responses and continued update for existing data
+	s := createServer(t, 8081)
+	go runServer(t, s)
+	defer s.ForceStop()
+
+	fileName := "../testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt"
+	routeTableDefaultRouteUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var routeTableDefaultRouteUpdateJson interface{}
+	json.Unmarshal(routeTableDefaultRouteUpdateByte, &routeTableDefaultRouteUpdateJson)
+
+	fileName = "../testdata/LLDP_ENTRY_TABLE_UPDATE.txt"
+	lldpEntryTableUpdateByte, err := ioutil.ReadFile(fileName)
+	if err != nil {
+		t.Fatalf("read file %v err: %v", fileName, err)
+	}
+	var lldpEntryTableUpdateJson interface{}
+	json.Unmarshal(lldpEntryTableUpdateByte, &lldpEntryTableUpdateJson)
+
+	tests := []struct {
+		desc     string
+		q        client.Query
+		wantNoti []client.Notification
+		poll     int
+	}{
+		{
+			desc: "query LLDP_ENTRY_TABLE + ROUTE_TABLE/0.0.0.0/0",
+			poll: 5,
+			q: client.Query{
+				Target:  "APPL_DB",
+				Type:    client.Poll,
+				Queries: []client.Path{{"LLDP_ENTRY_TABLE"}, {"ROUTE_TABLE", "0.0.0.0/0"}},
+				TLS:     &tls.Config{InsecureSkipVerify: true},
+			},
+			wantNoti: []client.Notification{
+				client.Connected{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Update{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200), Val: routeTableDefaultRouteUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Delete{Path: []string{"APPL_DB", "ROUTE_TABLE", "0.0.0.0/0"}, TS: time.Unix(0, 200)},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+				client.Update{Path: []string{"APPL_DB", "LLDP_ENTRY_TABLE"}, TS: time.Unix(0, 200), Val: lldpEntryTableUpdateJson},
+				client.Sync{},
+			},
+		},
+	}
+
+	ns, _ := sdcfg.GetDbDefaultNamespace()
+	rclient := getRedisClientN(t, 0, ns)
+	defer rclient.Close()
+	rclient.FlushDB()
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_port_id", "dummy")
+	rclient.HSet("LLDP_ENTRY_TABLE:eth0", "lldp_rem_sys_name", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "ifname", "dummy")
+	rclient.HSet("ROUTE_TABLE:0.0.0.0/0", "nexthop", "dummy")
+
+	var mutexGotNoti sync.Mutex
+
+	for _, tt := range tests {
+		t.Run(tt.desc, func(t *testing.T) {
+			q := tt.q
+			q.Addrs = []string{"127.0.0.1:8081"}
+			c := client.New()
+			var gotNoti []client.Notification
+
+			q.NotificationHandler = func(n client.Notification) error {
+				mutexGotNoti.Lock()
+				switch nn := n.(type) {
+				case client.Connected, client.Sync:
+					gotNoti = append(gotNoti, nn)
+				case client.Delete:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				case client.Update:
+					nn.TS = time.Unix(0, 200)
+					gotNoti = append(gotNoti, nn)
+				default:
+					t.Errorf("Unexpected Client Notification: %v", nn)
+				}
+				mutexGotNoti.Unlock()
+				return nil
+			}
+
+			wg := new(sync.WaitGroup)
+			wg.Add(1)
+
+			go func() {
+				defer wg.Done()
+				if err := c.Subscribe(context.Background(), q); err != nil {
+					t.Errorf("c.Subscribe(): got error %v, expected nil", err)
+				}
+			}()
+
+			wg.Wait()
+
+			for i := 0; i < tt.poll; i++ {
+				if i == 2 { // After first 2 polls, delete data
+					rclient.Del("ROUTE_TABLE:0.0.0.0/0")
+					// Sleep just one second to allow redis data to be deleted
+					time.Sleep(time.Millisecond * 1000)
+				}
+				err := c.Poll()
+				if err != nil {
+					t.Errorf("c.Poll(): got error %v, expected nil", err)
+				}
+			}
+
+			mutexGotNoti.Lock()
+
+			if len(gotNoti) == 0 || len(gotNoti) != len(tt.wantNoti) {
+				t.Errorf("Expected non zero length of notifications or equal notifications")
+			}
+
+			// Check that both notifications are coming at every poll interval
+			for _, pair := range [][2]int{{1, 2}, {4, 5}, {7, 8}, {10, 11}} { // these indexes are our update notifications
+				i, j := pair[0], pair[1]
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) || !reflect.DeepEqual(gotNoti[j], tt.wantNoti[j]) {
+					if !reflect.DeepEqual(gotNoti[j], tt.wantNoti[i]) && !reflect.DeepEqual(gotNoti[i], tt.wantNoti[j]) {
+						t.Fatalf("mismatch at indices %d/%d:\n got  (%#v, %#v)\n want (%#v, %#v)", i, j, gotNoti[i], gotNoti[j], tt.wantNoti[i], tt.wantNoti[j])
+					}
+				}
+			}
+
+			// check that the sync messages are identical
+			for i := 13; i < 17; i++ {
+				if !reflect.DeepEqual(gotNoti[i], tt.wantNoti[i]) {
+					t.Fatalf("notification %d mismatch:\n got  %#v\n want %#v", i, gotNoti[i], tt.wantNoti[i])
+				}
+			}
+
+			mutexGotNoti.Unlock()
+
+			c.Close()
+		})
+	}
+}
diff --git a/sonic_data_client/db_client.go b/sonic_data_client/db_client.go
index 1af08571c..88cc6d5e6 100644
--- a/sonic_data_client/db_client.go
+++ b/sonic_data_client/db_client.go
@@ -39,6 +39,8 @@ type Client interface {
 	// The service will stop upon detection of poll channel closing.
 	// It should run as a go routine
 	PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList)
+	// Poll to service AppDB only
+	AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList)
 	OnceRun(q *queue.PriorityQueue, once chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList)
 	// Get return data from the data source in format of *spb.Value
 	Get(w *sync.WaitGroup) ([]*spb.Value, error)
@@ -294,6 +296,69 @@ func streamSampleSubscription(c *DbClient, sub *gnmipb.Subscription, updateOnly
 	}
 }
 
+func (c *DbClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	c.w = w
+	defer c.w.Done()
+	c.q = q
+	c.channel = poll
+
+	prevUpdates := make(map[string]bool)
+
+	for {
+		_, more := <-c.channel
+		if !more {
+			log.V(1).Infof("%v poll channel closed, exiting pollDb routine", c)
+			return
+		}
+		t1 := time.Now()
+
+		for gnmiPath, tblPaths := range c.pathG2S {
+			pathKey := fmt.Sprintf("%v", gnmiPath)
+			val, err, updateReceived := AppDBTableData2TypedValue(tblPaths, nil)
+			if !updateReceived { // No updates sent for missing data
+				if prevUpdate, exists := prevUpdates[pathKey]; exists && prevUpdate == true {
+					log.V(6).Infof("Delete received for message for %v", gnmiPath)
+					spbv := &spb.Value{
+						Timestamp:    time.Now().UnixNano(),
+						Prefix:       c.prefix,
+						SyncResponse: false,
+						Delete:       []*gnmipb.Path{gnmiPath},
+						Val: &gnmipb.TypedValue{
+							Value: &gnmipb.TypedValue_StringVal{
+								StringVal: "",
+							},
+						},
+					}
+					c.q.Put(Value{spbv})
+					log.V(6).Infof("Added spbv #%v", spbv)
+					prevUpdates[pathKey] = false
+				}
+			} else if err != nil {
+				log.V(2).Infof("Unable to create gnmi TypedValue due to err: %v", err)
+				return
+			} else {
+				spbv := &spb.Value{
+					Prefix:       c.prefix,
+					Path:         gnmiPath,
+					Timestamp:    time.Now().UnixNano(),
+					SyncResponse: false,
+					Val:          val,
+				}
+				c.q.Put(Value{spbv})
+				prevUpdates[pathKey] = true
+				log.V(6).Infof("Added spbv #%v", spbv)
+			}
+		}
+		spbv := &spb.Value{
+			Timestamp:    time.Now().UnixNano(),
+			SyncResponse: true,
+		}
+		c.q.Put(Value{spbv})
+		log.V(6).Infof("Added spbv #%v", spbv)
+		log.V(4).Infof("Sync done, poll time taken: %v ms", int64(time.Since(t1)/time.Millisecond))
+	}
+}
+
 func (c *DbClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
 	c.w = w
 	defer c.w.Done()
@@ -334,6 +399,7 @@ func (c *DbClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.W
 		log.V(4).Infof("Sync done, poll time taken: %v ms", int64(time.Since(t1)/time.Millisecond))
 	}
 }
+
 func (c *DbClient) OnceRun(q *queue.PriorityQueue, once chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
 	return
 }
@@ -388,7 +454,6 @@ func ValToResp(val Value) (*gnmipb.SubscribeResponse, error) {
 				Response: &gnmipb.SubscribeResponse_Update{Update: n}}, nil
 		}
 
-		// In case of path deletion
 		if deleted := val.GetDelete(); deleted != nil {
 			return &gnmipb.SubscribeResponse{
 				Response: &gnmipb.SubscribeResponse_Update{
@@ -396,12 +461,6 @@ func ValToResp(val Value) (*gnmipb.SubscribeResponse, error) {
 						Timestamp: val.GetTimestamp(),
 						Prefix:    val.GetPrefix(),
 						Delete:    deleted,
-						Update: []*gnmipb.Update{
-							{
-								Path: val.GetPath(),
-								Val:  val.GetVal(),
-							},
-						},
 					},
 				},
 			}, nil
@@ -681,14 +740,34 @@ func populateDbtablePath(prefix, path *gnmipb.Path, pathG2S *map[*gnmipb.Path][]
 		log.V(6).Infof("Result of keys operation for %v %v, got %v", target, dbPath, res)
 		tblPath.tableKey = ""
 	case 3: // Third element could be table key; or field name in which case table name itself is the key too
-		n, err := redisDb.Exists(tblPath.tableName + tblPath.delimitor + mappedKey).Result()
-		if err != nil {
-			return fmt.Errorf("redis Exists op failed for %v", dbPath)
-		}
-		if n == 1 {
-			tblPath.tableKey = mappedKey
+		if targetDbName == "APPL_DB" {
+			keyExists, err := redisDb.Exists(tblPath.tableName + tblPath.delimitor + mappedKey).Result()
+			if err != nil {
+				return fmt.Errorf("redis Exists op failed for %v", dbPath)
+			}
+			if keyExists == 1 { // Existing Table:Key
+				tblPath.tableKey = mappedKey
+			} else {
+				fieldExists, err := redisDb.HExists(tblPath.tableName, mappedKey).Result()
+				if err != nil {
+					return fmt.Errorf("redis HExists op failed for %v", dbPath)
+				}
+				if fieldExists { // Existing field in Table
+					tblPath.field = mappedKey
+				} else { // Non existing table key
+					tblPath.tableKey = mappedKey
+				}
+			}
 		} else {
-			tblPath.field = mappedKey
+			n, err := redisDb.Exists(tblPath.tableName + tblPath.delimitor + mappedKey).Result()
+			if err != nil {
+				return fmt.Errorf("redis Exists op failed for %v", dbPath)
+			}
+			if n == 1 {
+				tblPath.tableKey = mappedKey
+			} else {
+				tblPath.field = mappedKey
+			}
 		}
 	case 4: // Fourth element could part of the table key or field name
 		tblPath.tableKey = mappedKey + tblPath.delimitor + stringSlice[3]
@@ -710,13 +789,15 @@ func populateDbtablePath(prefix, path *gnmipb.Path, pathG2S *map[*gnmipb.Path][]
 		return fmt.Errorf("Invalid db table Path %v", dbPath)
 	}
 
-	var key string
-	if tblPath.tableKey != "" {
-		key = tblPath.tableName + tblPath.delimitor + tblPath.tableKey
-		n, _ := redisDb.Exists(key).Result()
-		if n != 1 {
-			log.V(2).Infof("No valid entry found on %v with key %v", dbPath, key)
-			return fmt.Errorf("No valid entry found on %v with key %v", dbPath, key)
+	if targetDbName != "APPL_DB" {
+		var key string
+		if tblPath.tableKey != "" {
+			key = tblPath.tableName + tblPath.delimitor + tblPath.tableKey
+			n, _ := redisDb.Exists(key).Result()
+			if n != 1 {
+				log.V(2).Infof("No valid entry found on %v with key %v", dbPath, key)
+				return fmt.Errorf("No valid entry found on %v with key %v", dbPath, key)
+			}
 		}
 	}
 
@@ -847,6 +928,64 @@ func TableData2Msi(tblPath *tablePath, useKey bool, op *string, msi *map[string]
 	return nil
 }
 
+func AppDBTableData2Msi(tblPath *tablePath, useKey bool, op *string, msi *map[string]interface{}) error {
+	redisDb := Target2RedisDb[tblPath.dbNamespace][tblPath.dbName]
+
+	var pattern string
+	var dbkeys []string
+	var err error
+	var fv map[string]string
+
+	//Only table name provided
+	if tblPath.tableKey == "" {
+		// tables in COUNTERS_DB other than COUNTERS table doesn't have keys
+		pattern = tblPath.tableName + tblPath.delimitor + "*"
+		dbkeys, err = redisDb.Keys(pattern).Result()
+		if err != nil {
+			log.V(2).Infof("redis Keys failed for %v, pattern %s", tblPath, pattern)
+			return fmt.Errorf("redis Keys failed for %v, pattern %s %v", tblPath, pattern, err)
+		}
+	} else {
+		// both table name and key provided
+		dbkeys = []string{tblPath.tableName + tblPath.delimitor + tblPath.tableKey}
+	}
+
+	log.V(4).Infof("dbkeys to be pulled from redis %v", dbkeys)
+
+	for idx, dbkey := range dbkeys {
+		fv, err = redisDb.HGetAll(dbkey).Result()
+		if err != nil {
+			log.V(2).Infof("redis HGetAll failed for  %v, dbkey %s", tblPath, dbkey)
+			return err
+		}
+		log.V(4).Infof("Data pulled for dbkey %s: %v", dbkey, fv)
+
+		if len(fv) == 0 { // Skip update for non data path
+			log.V(6).Infof("Missing data for dbkey %s, will check next key", dbkey)
+			continue
+		}
+
+		if (tblPath.tableKey != "" && !useKey) || tblPath.tableName == dbkey {
+			err = makeJSON_redis(msi, nil, op, fv)
+		} else {
+			var key string
+			// Split dbkey string into two parts and second part is key in table
+			keys := strings.SplitN(dbkey, tblPath.delimitor, 2)
+			if len(keys) < 2 {
+				return fmt.Errorf("dbkey: %s, failed split from delimitor %v", dbkey, tblPath.delimitor)
+			}
+			key = keys[1]
+			err = makeJSON_redis(msi, &key, op, fv)
+		}
+		if err != nil {
+			log.V(2).Infof("makeJSON err %s for fv %v", err, fv)
+			return err
+		}
+		log.V(6).Infof("Added idex %v fv %v ", idx, fv)
+	}
+	return nil
+}
+
 func Msi2TypedValue(msi map[string]interface{}) (*gnmipb.TypedValue, error) {
 	log.V(4).Infof("State of map after adding redis data %v", msi)
 	jv, err := emitJSON(&msi)
@@ -903,6 +1042,59 @@ func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue,
 	return Msi2TypedValue(msi)
 }
 
+// Returns typed value, error, or bool. Bool specifies that there is data available for the tablePath that was queried.
+func AppDBTableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error, bool) {
+	var useKey bool
+	var updateReceived bool
+	msi := make(map[string]interface{})
+	for _, tblPath := range tblPaths {
+		redisDb := Target2RedisDb[tblPath.dbNamespace][tblPath.dbName]
+
+		if tblPath.jsonField == "" { // Not asked to include field in json value, which means not wildcard query
+			// table path includes table, key and field
+			if tblPath.field != "" {
+				if len(tblPaths) != 1 {
+					log.V(2).Infof("WARNING: more than one path exists for field granularity query: %v", tblPaths)
+				}
+				var key string
+				if tblPath.tableKey != "" {
+					key = tblPath.tableName + tblPath.delimitor + tblPath.tableKey
+				} else {
+					key = tblPath.tableName
+				}
+
+				val, err := redisDb.HGet(key, tblPath.field).Result()
+				if err != nil {
+					log.V(2).Infof("redis HGet failed for %v, data does not exist", tblPath)
+					continue
+				}
+				log.V(4).Infof("Data pulled for key %s and field %s: %s", key, tblPath.field, val)
+				// TODO: support multiple table paths
+				return &gnmipb.TypedValue{
+					Value: &gnmipb.TypedValue_StringVal{
+						StringVal: val,
+					}}, nil, true
+			}
+		}
+		err := AppDBTableData2Msi(&tblPath, useKey, nil, &msi)
+
+		if err != nil {
+			return nil, err, true
+		}
+
+		if !updateReceived {
+			if len(msi) > 0 { // Update occurred
+				updateReceived = true
+			}
+		}
+	}
+	if !updateReceived {
+		return nil, nil, false
+	}
+	val, err := Msi2TypedValue(msi)
+	return val, err, true
+}
+
 func enqueueFatalMsg(c *DbClient, msg string) {
 	putFatalMsg(c.q, msg)
 }
@@ -1149,7 +1341,6 @@ func dbSingleTableKeySubscribe(c *DbClient, rsd redisSubData, updateChannel chan
 					}
 					key := subscr.Channel[prefixLen:]
 					newMsi[key] = fp
-					newMsi["delete"] = "null_value"
 				}
 			} else if subscr.Payload == "hset" {
 				//op := "SET"
@@ -1221,11 +1412,6 @@ func dbTableKeySubscribe(c *DbClient, gnmiPath *gnmipb.Path, interval time.Durat
 
 	// Helper to send hash data over the stream
 	sendMsiData := func(msiData map[string]interface{}) error {
-		sendDeleteField := false
-		if _, isDelete := msiData["delete"]; isDelete {
-			sendDeleteField = true
-		}
-		delete(msiData, "delete")
 		val, err := Msi2TypedValue(msiData)
 		if err != nil {
 			return err
@@ -1238,9 +1424,6 @@ func dbTableKeySubscribe(c *DbClient, gnmiPath *gnmipb.Path, interval time.Durat
 			Timestamp: time.Now().UnixNano(),
 			Val:       val,
 		}
-		if sendDeleteField {
-			(*spbv).Delete = []*gnmipb.Path{gnmiPath}
-		}
 		if err = c.q.Put(Value{spbv}); err != nil {
 			return fmt.Errorf("Queue error:  %v", err)
 		}
diff --git a/sonic_data_client/events_client.go b/sonic_data_client/events_client.go
index 5086c2976..a8c6b03d5 100644
--- a/sonic_data_client/events_client.go
+++ b/sonic_data_client/events_client.go
@@ -449,6 +449,10 @@ func (evtc *EventClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, wg
 	return
 }
 
+func (evtc *EventClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, wg *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	return
+}
+
 func (evtc *EventClient) Close() error {
 	return nil
 }
diff --git a/sonic_data_client/mixed_db_client.go b/sonic_data_client/mixed_db_client.go
index 1a8212c7a..f2ef3cbf1 100644
--- a/sonic_data_client/mixed_db_client.go
+++ b/sonic_data_client/mixed_db_client.go
@@ -1606,6 +1606,10 @@ func (c *MixedDbClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *s
 	}
 }
 
+func (c *MixedDbClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	return
+}
+
 func (c *MixedDbClient) StreamRun(q *queue.PriorityQueue, stop chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
 	c.w = w
 	defer c.w.Done()
diff --git a/sonic_data_client/non_db_client.go b/sonic_data_client/non_db_client.go
index d0bc847d5..5b3af9862 100644
--- a/sonic_data_client/non_db_client.go
+++ b/sonic_data_client/non_db_client.go
@@ -553,6 +553,11 @@ func (c *NonDbClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *syn
 		log.V(4).Infof("Sync done, poll time taken: %v ms", int64(time.Since(t1)/time.Millisecond))
 	}
 }
+
+func (c *NonDbClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	return
+}
+
 func (c *NonDbClient) OnceRun(q *queue.PriorityQueue, once chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
 	return
 }
diff --git a/sonic_data_client/transl_data_client.go b/sonic_data_client/transl_data_client.go
index 21e639253..da545ab87 100644
--- a/sonic_data_client/transl_data_client.go
+++ b/sonic_data_client/transl_data_client.go
@@ -415,6 +415,10 @@ func (c *TranslClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sy
 	}
 }
 
+func (c *TranslClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	return
+}
+
 func (c *TranslClient) OnceRun(q *queue.PriorityQueue, once chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
 	c.w = w
 	defer c.w.Done()
diff --git a/testdata/LLDP_ENTRY_TABLE_UPDATE.txt b/testdata/LLDP_ENTRY_TABLE_UPDATE.txt
new file mode 100644
index 000000000..ceae6879b
--- /dev/null
+++ b/testdata/LLDP_ENTRY_TABLE_UPDATE.txt
@@ -0,0 +1,6 @@
+{
+    "eth0": {
+        "lldp_rem_port_id": "dummy",
+        "lldp_rem_sys_name": "dummy"
+    }
+}
diff --git a/testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt b/testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt
new file mode 100644
index 000000000..19092360a
--- /dev/null
+++ b/testdata/ROUTE_TABLE_DEFAULT_ROUTE_UPDATE.txt
@@ -0,0 +1,4 @@
+{
+    "ifname": "dummy",
+    "nexthop": "dummy"
+}


Reviews:
user: login: copilot-pull-request-reviewer[bot]
body: 

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.


<details>
<summary>Comments suppressed due to low confidence (1)</summary>

**sonic_data_client/db_client.go:317**
* The logic for handling noUpdate relies on the prevUpdates map but may not cover all edge cases when table data is missing. Consider implementing the deletion marker as noted by the TODO comment to clearly signal deletions and avoid ambiguity.
```
if prevUpdate, exists := prevUpdates[pathKey]; exists && prevUpdate == true {
```
</details>


state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T15:14:18Z
user: login: qiluo-msft
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-15T15:18:40Z
user: login: hdwhdw
body: Can you explain the background of the change. It looks like to me that we are adding entries whenever someone poll an nonexistent table or key. Is this okay? Shouldn't failure be the default option here?
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T18:55:01Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T20:00:55Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T20:03:52Z
user: login: hdwhdw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T20:52:30Z
user: login: hdwhdw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T21:05:57Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T21:43:13Z
user: login: qiluo-msft
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T06:49:59Z
user: login: qiluo-msft
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-17T06:56:00Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T16:53:30Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-17T23:01:59Z
user: login: qiluo-msft
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T06:04:40Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T06:17:33Z
user: login: qiluo-msft
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-04-18T06:50:50Z
user: login: FengPan-Frank
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-04-18T07:23:09Z
user: login: hdwhdw
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T15:37:29Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:12:16Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:18:08Z
user: login: zbud-msft
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:19:39Z
user: login: hdwhdw
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-18T16:29:12Z

Review comments:
diff_hunk: @@ -681,14 +703,22 @@ func populateDbtablePath(prefix, path *gnmipb.Path, pathG2S *map[*gnmipb.Path][]
 		log.V(6).Infof("Result of keys operation for %v %v, got %v", target, dbPath, res)
 		tblPath.tableKey = ""
 	case 3: // Third element could be table key; or field name in which case table name itself is the key too
-		n, err := redisDb.Exists(tblPath.tableName + tblPath.delimitor + mappedKey).Result()
+		keyExists, err := redisDb.Exists(tblPath.tableName + tblPath.delimitor + mappedKey).Result()
path: sonic_data_client/db_client.go
user: login: Copilot
body: [nitpick] Consider renaming 'delimitor' to 'delimiter' for correct spelling and improved clarity.
```suggestion
		keyExists, err := redisDb.Exists(tblPath.tableName + tblPath.delimiter + mappedKey).Result()
```
created_at: 2025-04-15T15:14:18Z
author_association: CONTRIBUTOR
diff_hunk: @@ -300,37 +300,59 @@ func (c *DbClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.W
 	c.q = q
 	c.channel = poll
 
+	prevUpdates := make(map[string]bool)
path: sonic_data_client/db_client.go
user: login: qiluo-msft
body: >prevUpdates [](http://example.com/codeflow?start=1&length=11)

This map will always increase size in lifetime of telemetry process. Why you need it? #Closed
created_at: 2025-04-15T15:18:39Z
author_association: COLLABORATOR
diff_hunk: @@ -863,7 +888,7 @@ func Msi2TypedValue(msi map[string]interface{}) (*gnmipb.TypedValue, error) {
 		}}, nil
 }
 
-func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error) {
+func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error, bool) {
path: sonic_data_client/db_client.go
user: login: hdwhdw
body: Can you leave some documentation on what exactly is this bool? Reader might want to check whether the logic around this bool is sound but it is not possible here.
created_at: 2025-04-15T18:46:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -884,23 +909,28 @@ func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue,
 
 				val, err := redisDb.HGet(key, tblPath.field).Result()
 				if err != nil {
-					log.V(2).Infof("redis HGet failed for %v", tblPath)
-					return nil, err
+					log.V(2).Infof("redis HGet failed for %v, data does not exist", tblPath)
+					return nil, err, true
 				}
 				log.V(4).Infof("Data pulled for key %s and field %s: %s", key, tblPath.field, val)
 				// TODO: support multiple table paths
 				return &gnmipb.TypedValue{
 					Value: &gnmipb.TypedValue_StringVal{
 						StringVal: val,
-					}}, nil
+					}}, nil, false
 			}
 		}
 		err := TableData2Msi(&tblPath, useKey, nil, &msi)
+
 		if err != nil {
-			return nil, err
+			return nil, err, false
+		}
+		if len(msi) == 0 {
+			return nil, nil, true
path: sonic_data_client/db_client.go
user: login: hdwhdw
body: Question: Why do we want to return early here?
created_at: 2025-04-15T18:48:11Z
author_association: CONTRIBUTOR
diff_hunk: @@ -863,7 +888,7 @@ func Msi2TypedValue(msi map[string]interface{}) (*gnmipb.TypedValue, error) {
 		}}, nil
 }
 
-func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error) {
+func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error, bool) {
path: sonic_data_client/db_client.go
user: login: zbud-msft
body: Added comment.
created_at: 2025-04-15T20:00:55Z
author_association: CONTRIBUTOR
diff_hunk: @@ -884,23 +909,28 @@ func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue,
 
 				val, err := redisDb.HGet(key, tblPath.field).Result()
 				if err != nil {
-					log.V(2).Infof("redis HGet failed for %v", tblPath)
-					return nil, err
+					log.V(2).Infof("redis HGet failed for %v, data does not exist", tblPath)
+					return nil, err, true
 				}
 				log.V(4).Infof("Data pulled for key %s and field %s: %s", key, tblPath.field, val)
 				// TODO: support multiple table paths
 				return &gnmipb.TypedValue{
 					Value: &gnmipb.TypedValue_StringVal{
 						StringVal: val,
-					}}, nil
+					}}, nil, false
 			}
 		}
 		err := TableData2Msi(&tblPath, useKey, nil, &msi)
+
 		if err != nil {
-			return nil, err
+			return nil, err, false
+		}
+		if len(msi) == 0 {
+			return nil, nil, true
path: sonic_data_client/db_client.go
user: login: zbud-msft
body: We want to return early here because since there are no updates for the path that was queried, we can return back to PollRun and send a sync response only (data has been missing for more than a single poll interval) or a delete notification (data is missing since last poll interval)
created_at: 2025-04-15T20:03:52Z
author_association: CONTRIBUTOR
diff_hunk: @@ -863,7 +888,7 @@ func Msi2TypedValue(msi map[string]interface{}) (*gnmipb.TypedValue, error) {
 		}}, nil
 }
 
-func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error) {
+func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error, bool) {
path: sonic_data_client/db_client.go
user: login: hdwhdw
body: Thanks. Also: negative predicate is in general discouraged because it creates confusion for readers and is error prone. i.e. API like hasData() is much better than NoData().  Consider negating this definition.
created_at: 2025-04-15T20:51:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -884,23 +909,28 @@ func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue,
 
 				val, err := redisDb.HGet(key, tblPath.field).Result()
 				if err != nil {
-					log.V(2).Infof("redis HGet failed for %v", tblPath)
-					return nil, err
+					log.V(2).Infof("redis HGet failed for %v, data does not exist", tblPath)
+					return nil, err, true
 				}
 				log.V(4).Infof("Data pulled for key %s and field %s: %s", key, tblPath.field, val)
 				// TODO: support multiple table paths
 				return &gnmipb.TypedValue{
 					Value: &gnmipb.TypedValue_StringVal{
 						StringVal: val,
-					}}, nil
+					}}, nil, false
 			}
 		}
 		err := TableData2Msi(&tblPath, useKey, nil, &msi)
+
 		if err != nil {
-			return nil, err
+			return nil, err, false
+		}
+		if len(msi) == 0 {
+			return nil, nil, true
path: sonic_data_client/db_client.go
user: login: hdwhdw
body: Thanks!
created_at: 2025-04-15T21:05:57Z
author_association: CONTRIBUTOR
diff_hunk: @@ -863,7 +888,7 @@ func Msi2TypedValue(msi map[string]interface{}) (*gnmipb.TypedValue, error) {
 		}}, nil
 }
 
-func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error) {
+func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error, bool) {
path: sonic_data_client/db_client.go
user: login: zbud-msft
body: Agree, will change.
created_at: 2025-04-15T21:43:13Z
author_association: CONTRIBUTOR
diff_hunk: @@ -553,6 +553,11 @@ func (c *NonDbClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *syn
 		log.V(4).Infof("Sync done, poll time taken: %v ms", int64(time.Since(t1)/time.Millisecond))
 	}
 }
+
+func (c *NonDbClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
path: sonic_data_client/non_db_client.go
user: login: qiluo-msft
body: > { [](http://example.com/codeflow?start=133&length=2)

Please fix the indentation. #Closed
created_at: 2025-04-17T06:49:59Z
author_association: COLLABORATOR
diff_hunk: @@ -39,6 +39,8 @@ type Client interface {
 	// The service will stop upon detection of poll channel closing.
 	// It should run as a go routine
 	PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList)
+	// Poll to service AppDB only
+	AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList)
path: sonic_data_client/db_client.go
user: login: qiluo-msft
body: >AppDBPollRun [](http://example.com/codeflow?start=1&length=12)

I could not find the code calling it, do you miss some file? #Closed
created_at: 2025-04-17T06:56:00Z
author_association: COLLABORATOR
diff_hunk: @@ -39,6 +39,8 @@ type Client interface {
 	// The service will stop upon detection of poll channel closing.
 	// It should run as a go routine
 	PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList)
+	// Poll to service AppDB only
+	AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList)
path: sonic_data_client/db_client.go
user: login: zbud-msft
body: Fixed
created_at: 2025-04-17T16:53:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -863,7 +888,7 @@ func Msi2TypedValue(msi map[string]interface{}) (*gnmipb.TypedValue, error) {
 		}}, nil
 }
 
-func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error) {
+func tableData2TypedValue(tblPaths []tablePath, op *string) (*gnmipb.TypedValue, error, bool) {
path: sonic_data_client/db_client.go
user: login: zbud-msft
body: Fixed.
created_at: 2025-04-17T23:01:59Z
author_association: CONTRIBUTOR
diff_hunk: @@ -195,7 +195,11 @@ func (c *Client) Run(stream gnmipb.GNMI_SubscribeServer) (err error) {
 		c.polled = make(chan struct{}, 1)
 		c.polled <- struct{}{}
 		c.w.Add(1)
-		go dc.PollRun(c.q, c.polled, &c.w, c.subscribe)
+		if strings.Contains(target, "APPL_DB") {
path: gnmi_server/client_subscribe.go
user: login: qiluo-msft
body: >strings.Contains [](http://example.com/codeflow?start=5&length=16)

too relaxing condition, please be more specific, check the single-asic or multi-asic APPL_DB only. #Closed
created_at: 2025-04-18T06:04:40Z
author_association: COLLABORATOR
diff_hunk: @@ -195,7 +195,11 @@ func (c *Client) Run(stream gnmipb.GNMI_SubscribeServer) (err error) {
 		c.polled = make(chan struct{}, 1)
 		c.polled <- struct{}{}
 		c.w.Add(1)
-		go dc.PollRun(c.q, c.polled, &c.w, c.subscribe)
+		if strings.Contains(target, "APPL_DB") {
path: gnmi_server/client_subscribe.go
user: login: zbud-msft
body: Done
created_at: 2025-04-18T06:17:33Z
author_association: CONTRIBUTOR
diff_hunk: @@ -195,7 +195,11 @@ func (c *Client) Run(stream gnmipb.GNMI_SubscribeServer) (err error) {
 		c.polled = make(chan struct{}, 1)
 		c.polled <- struct{}{}
 		c.w.Add(1)
-		go dc.PollRun(c.q, c.polled, &c.w, c.subscribe)
+		if target == "APPL_DB" || strings.HasPrefix(target, "APPL_DB/") {
path: gnmi_server/client_subscribe.go
user: login: FengPan-Frank
body: is this only applicable for APPL_DB?
created_at: 2025-04-18T07:23:09Z
author_association: COLLABORATOR
diff_hunk: @@ -294,6 +296,69 @@ func streamSampleSubscription(c *DbClient, sub *gnmipb.Subscription, updateOnly
 	}
 }
 
+func (c *DbClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	c.w = w
+	defer c.w.Done()
+	c.q = q
+	c.channel = poll
+
+	prevUpdates := make(map[string]bool)
+
+	for {
+		_, more := <-c.channel
+		if !more {
+			log.V(1).Infof("%v poll channel closed, exiting pollDb routine", c)
+			return
+		}
+		t1 := time.Now()
+
+		for gnmiPath, tblPaths := range c.pathG2S {
+			pathKey := fmt.Sprintf("%v", gnmiPath)
+			val, err, updateReceived := AppDBTableData2TypedValue(tblPaths, nil)
+			if !updateReceived { // No updates sent for missing data
path: sonic_data_client/db_client.go
user: login: hdwhdw
body: Is this comment stale?
created_at: 2025-04-18T15:36:40Z
author_association: CONTRIBUTOR
diff_hunk: @@ -195,7 +195,11 @@ func (c *Client) Run(stream gnmipb.GNMI_SubscribeServer) (err error) {
 		c.polled = make(chan struct{}, 1)
 		c.polled <- struct{}{}
 		c.w.Add(1)
-		go dc.PollRun(c.q, c.polled, &c.w, c.subscribe)
+		if target == "APPL_DB" || strings.HasPrefix(target, "APPL_DB/") {
path: gnmi_server/client_subscribe.go
user: login: zbud-msft
body: For now just APP DB, will integrate these changes for all DB as a TODO
created_at: 2025-04-18T16:12:16Z
author_association: CONTRIBUTOR
diff_hunk: @@ -294,6 +296,69 @@ func streamSampleSubscription(c *DbClient, sub *gnmipb.Subscription, updateOnly
 	}
 }
 
+func (c *DbClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	c.w = w
+	defer c.w.Done()
+	c.q = q
+	c.channel = poll
+
+	prevUpdates := make(map[string]bool)
+
+	for {
+		_, more := <-c.channel
+		if !more {
+			log.V(1).Infof("%v poll channel closed, exiting pollDb routine", c)
+			return
+		}
+		t1 := time.Now()
+
+		for gnmiPath, tblPaths := range c.pathG2S {
+			pathKey := fmt.Sprintf("%v", gnmiPath)
+			val, err, updateReceived := AppDBTableData2TypedValue(tblPaths, nil)
+			if !updateReceived { // No updates sent for missing data
path: sonic_data_client/db_client.go
user: login: zbud-msft
body: No, I changed from NoUpdate (where I used negation wording) to !UpdateReceived, since we now return true if we have data versus true if we have no data.
created_at: 2025-04-18T16:18:07Z
author_association: CONTRIBUTOR
diff_hunk: @@ -195,7 +195,11 @@ func (c *Client) Run(stream gnmipb.GNMI_SubscribeServer) (err error) {
 		c.polled = make(chan struct{}, 1)
 		c.polled <- struct{}{}
 		c.w.Add(1)
-		go dc.PollRun(c.q, c.polled, &c.w, c.subscribe)
+		if target == "APPL_DB" || strings.HasPrefix(target, "APPL_DB/") {
path: gnmi_server/client_subscribe.go
user: login: zbud-msft
body: Issue: https://github.com/sonic-net/sonic-gnmi/issues/379
created_at: 2025-04-18T16:19:39Z
author_association: CONTRIBUTOR
diff_hunk: @@ -294,6 +296,69 @@ func streamSampleSubscription(c *DbClient, sub *gnmipb.Subscription, updateOnly
 	}
 }
 
+func (c *DbClient) AppDBPollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {
+	c.w = w
+	defer c.w.Done()
+	c.q = q
+	c.channel = poll
+
+	prevUpdates := make(map[string]bool)
+
+	for {
+		_, more := <-c.channel
+		if !more {
+			log.V(1).Infof("%v poll channel closed, exiting pollDb routine", c)
+			return
+		}
+		t1 := time.Now()
+
+		for gnmiPath, tblPaths := range c.pathG2S {
+			pathKey := fmt.Sprintf("%v", gnmiPath)
+			val, err, updateReceived := AppDBTableData2TypedValue(tblPaths, nil)
+			if !updateReceived { // No updates sent for missing data
path: sonic_data_client/db_client.go
user: login: hdwhdw
body: Ah thanks!
created_at: 2025-04-18T16:29:12Z
author_association: CONTRIBUTOR

PR comments:
user: login: mssonicbld
created_at: 2025-04-15T07:51:24Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-15T07:51:34Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: qiluo-msft
created_at: 2025-04-15T15:20:42Z
body: > func (c *DbClient) PollRun(q *queue.PriorityQueue, poll chan struct{}, w *sync.WaitGroup, subscribe *gnmipb.SubscriptionList) {

Could you add testcases which behavior different on old and new code? #Closed

---
Refers to: sonic_data_client/db_client.go:297 in 6e6c3ca. [](commit_id = 6e6c3cac7b2ce54f506a59817f92635a53189981, deletion_comment = False)
author_association: COLLABORATOR
user: login: mssonicbld
created_at: 2025-04-15T18:54:34Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-15T18:54:43Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-15T20:00:41Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-15T20:00:50Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: zbud-msft
created_at: 2025-04-15T20:06:46Z
body: > Can you explain the background of the change. It looks like to me that we are adding entries whenever someone poll an nonexistent table or key. Is this okay? Shouldn't failure be the default option here?

Updated the PR description with more details. We are sending sync responses only to keep the RPC available and to let the client know that there is no data available but we will send data when it becomes available. Failure and closing RCP connection (current behavior) does not conform to official gnmi spec
author_association: CONTRIBUTOR
user: login: mssonicbld
created_at: 2025-04-16T21:40:29Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-16T21:40:39Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-16T21:58:19Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-16T21:58:28Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-17T05:03:31Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-17T05:03:40Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-17T07:21:29Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-17T07:21:40Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: zbud-msft
created_at: 2025-04-17T16:54:10Z
body: /azp run

author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-17T16:54:19Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-17T22:46:28Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-17T22:46:37Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-17T23:01:27Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-17T23:01:36Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: qiluo-msft
created_at: 2025-04-17T23:16:41Z
body: LGTM. Please test on both single-asic, T2 multi-asic DUTs. #Closed
author_association: COLLABORATOR
user: login: mssonicbld
created_at: 2025-04-18T01:26:42Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-18T01:26:51Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-18T01:45:08Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-18T01:45:18Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-18T05:26:59Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-18T05:27:08Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-18T06:12:38Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-18T06:12:48Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: mssonicbld
created_at: 2025-04-18T06:21:37Z
body: /azp run
author_association: CONTRIBUTOR
user: login: azure-pipelines[bot]
created_at: 2025-04-18T06:21:47Z
body: <samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
author_association: NONE
user: login: r12f
created_at: 2025-06-28T16:46:41Z
body: merged in 202411. fixing tag.
author_association: NONE