Repository: edgexfoundry/device-sdk-go
Pull request #1672 (2025-01-07 12:52:13)

feat: Add device up/down detection
fix: #1665 

<!-- Expected Commit Message Description (imported automatically by GitHub) -->
<!-- Must conform to [conventional commits guidelines](https://github.com/edgexfoundry/device-sdk-go/blob/main/.github/Contributing.md) -->
<!-- Expected Commit message must contain Closes/Fixes #IssueNumber statement when there is a related issue -->

<!-- Add additional detailed description of need for change if no related issue -->

## PR Checklist
Please check if your PR fulfills the following requirements:

- [x] I am not introducing a breaking change (if you are, flag in conventional commit message with `BREAKING CHANGE:` describing the break)
- [x] I am not introducing a new dependency (add notes below if you are)
- [ ] I have added unit tests for the new feature or bug fix (if not, why?)
- [x] I have fully tested (add details below) this the new feature or bug fix (if not, why?)
- [x] I have opened a PR for the related docs change (if not, why?) 
  https://github.com/edgexfoundry/edgex-docs/pull/1407

## Testing Instructions
<!-- How can the reviewers test your change? -->
Run Edgex stack using Edgex compose-builder
```
make run no-secty modbus-sim
```
Run device-modbus with this version of the device sdk and the following configuration
```
Writable:
  LogLevel: INFO

Service:
  Host: localhost
  Port: 59901
  StartupMsg: device modbus started

MessageBus:
  Optional:
    ClientId: device-modbus

Device:
  ProfilesDir: ""
  DevicesDir: ""
  AllowedFails: 1
  DeviceDownTimeout: 10
```
Upload device profile and onboard a device
```
# profile sample: https://github.com/edgexfoundry/device-modbus-go/blob/main/cmd/res/profiles/modbus.test.device.profile.yml 
curl http://localhost:59881/api/v3/deviceprofile/uploadfile -F "file=@modbus.test.device.profile.yml"
```
```
curl http://localhost:59881/api/v3/device -H "Content-Type:application/json" -X POST \
  -d '[
        {
            "apiVersion" : "v3",
            "device": {
               "name" :"Modbus-TCP-Temperature-Sensor",
               "description":"This device is a product for monitoring the temperature via the ethernet",
               "labels":[ 
                  "Temperature",
                  "Modbus TCP"
               ],
               "serviceName": "device-modbus",
               "profileName": "Test-Device-Modbus-Profile",
               "protocols":{
                  "modbus-tcp":{
                     "Address" : "localhost",
                     "Port" : "1502",
                     "UnitID" : "1",
                     "Timeout" : "1",
                     "IdleTimeout" : "1"
                  }
               },
               "autoEvents":[ 
                  { 
                     "Interval":"5s",
                     "onChange":false,
                     "SourceName":"Temperature"
                  }
               ],
               "adminState":"UNLOCKED",
               "operatingState":"UP"
            }
        }
    ]'
```
Stop the modbus simulator and the following will be in device-modbus's log
```
level=INFO ts=2025-01-07T20:23:55.013483+08:00 app=device-modbus source=driver.go:137 msg="Read command failed. Cmd:Temperature err:dial tcp [::1]:1502: connect: connection refused \n"
level=INFO ts=2025-01-07T20:23:55.013616+08:00 app=device-modbus source=opstate.go:85 msg="Marking device Modbus-TCP-Temperature-Sensor non-operational"
level=WARN ts=2025-01-07T20:23:55.036664+08:00 app=device-modbus source=opstate.go:88 msg="Will retry device Modbus-TCP-Temperature-Sensor in 10 seconds"
level=ERROR ts=2025-01-07T20:23:55.036749+08:00 app=device-modbus source=executor.go:61 msg="AutoEvent - error occurs when reading resource Temperature: error reading Regex DeviceResource(s) Temperature for Modbus-TCP-Temperature-Sensor -> dial tcp [::1]:1502: connect: connection refused"

level=INFO ts=2025-01-07T20:24:05.037492+08:00 app=device-modbus source=opstate.go:31 msg="Checking operational state for device: Modbus-TCP-Temperature-Sensor"
level=INFO ts=2025-01-07T20:24:05.040394+08:00 app=device-modbus source=driver.go:137 msg="Read command failed. Cmd:SwitchA err:dial tcp [::1]:1502: connect: connection refused \n"
level=ERROR ts=2025-01-07T20:24:05.040509+08:00 app=device-modbus source=opstate.go:60 msg="Device Modbus-TCP-Temperature-Sensor unresponsive: retrying in 10 seconds."
```
Restart the modbus simulator and the following will be in device-modbus's log
```
level=INFO ts=2025-01-07T20:24:25.04497+08:00 app=device-modbus source=opstate.go:31 msg="Checking operational state for device: Modbus-TCP-Temperature-Sensor"
2025/01/07 20:24:25 modbus: sending 00 08 00 00 00 06 01 01 00 00 00 01
2025/01/07 20:24:25 modbus: received 00 08 00 00 00 04 01 01 01 00
level=INFO ts=2025-01-07T20:24:25.08558+08:00 app=device-modbus source=modbusclient.go:91 msg="Modbus client GetValue's results [0]"
level=INFO ts=2025-01-07T20:24:25.086155+08:00 app=device-modbus source=driver.go:167 msg="Read command finished. Cmd:SwitchA, DeviceResource: SwitchA, Bool: false \n"
level=INFO ts=2025-01-07T20:24:25.086872+08:00 app=device-modbus source=opstate.go:56 msg="Device Modbus-TCP-Temperature-Sensor responsive: setting operational state to up."

```
Or if the device's operating state has already returned to UP before the check, the following will be in device-modbus's log
```
level=INFO ts=2025-01-07T20:29:09.03874+08:00 app=device-modbus source=opstate.go:31 msg="Checking operational state for device: Modbus-TCP-Temperature-Sensor"
level=INFO ts=2025-01-07T20:29:09.03894+08:00 app=device-modbus source=opstate.go:40 msg="Device Modbus-TCP-Temperature-Sensor is already operational. Exiting retry loop."
```





Patch:
diff --git a/go.mod b/go.mod
index a600a7cb..f392e7ca 100644
--- a/go.mod
+++ b/go.mod
@@ -4,7 +4,7 @@ go 1.23
 
 require (
 	github.com/OneOfOne/xxhash v1.2.8
-	github.com/edgexfoundry/go-mod-bootstrap/v4 v4.0.0-dev.17
+	github.com/edgexfoundry/go-mod-bootstrap/v4 v4.0.0-dev.16
 	github.com/edgexfoundry/go-mod-core-contracts/v4 v4.0.0-dev.19
 	github.com/edgexfoundry/go-mod-messaging/v4 v4.0.0-dev.10
 	github.com/google/uuid v1.6.0
@@ -14,7 +14,7 @@ require (
 	github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
 	github.com/spf13/cast v1.7.1
 	github.com/stretchr/testify v1.10.0
-	golang.org/x/net v0.34.0
+	golang.org/x/net v0.33.0
 	gopkg.in/yaml.v3 v3.0.1
 )
 
@@ -112,12 +112,12 @@ require (
 	go.opentelemetry.io/otel v1.29.0 // indirect
 	go.opentelemetry.io/otel/metric v1.29.0 // indirect
 	go.opentelemetry.io/otel/trace v1.29.0 // indirect
-	golang.org/x/crypto v0.32.0 // indirect
+	golang.org/x/crypto v0.31.0 // indirect
 	golang.org/x/exp v0.0.0-20230817173708-d852ddb80c63 // indirect
 	golang.org/x/oauth2 v0.23.0 // indirect
 	golang.org/x/sync v0.10.0 // indirect
-	golang.org/x/sys v0.29.0 // indirect
-	golang.org/x/term v0.28.0 // indirect
+	golang.org/x/sys v0.28.0 // indirect
+	golang.org/x/term v0.27.0 // indirect
 	golang.org/x/text v0.21.0 // indirect
 	golang.org/x/time v0.8.0 // indirect
 	google.golang.org/genproto/googleapis/rpc v0.0.0-20240814211410-ddb44dafa142 // indirect
diff --git a/go.sum b/go.sum
index f30efbce..415e1b48 100644
--- a/go.sum
+++ b/go.sum
@@ -70,8 +70,8 @@ github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1
 github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/eclipse/paho.mqtt.golang v1.5.0 h1:EH+bUVJNgttidWFkLLVKaQPGmkTUfQQqjOsyvMGvD6o=
 github.com/eclipse/paho.mqtt.golang v1.5.0/go.mod h1:du/2qNQVqJf/Sqs4MEL77kR8QTqANF7XU7Fk0aOTAgk=
-github.com/edgexfoundry/go-mod-bootstrap/v4 v4.0.0-dev.17 h1:0cMD8/A1vVg2u7igPO35/spG8bZd//dHBXOlspArWyk=
-github.com/edgexfoundry/go-mod-bootstrap/v4 v4.0.0-dev.17/go.mod h1:66I+qRA22YkjA/SYw4F9R0avZ9oNYM8bg2qhwePTFkI=
+github.com/edgexfoundry/go-mod-bootstrap/v4 v4.0.0-dev.16 h1:NSEoo/YnSDNWQcZpLAtXqDKHAId5mecTVBxk9zs/ROg=
+github.com/edgexfoundry/go-mod-bootstrap/v4 v4.0.0-dev.16/go.mod h1:66I+qRA22YkjA/SYw4F9R0avZ9oNYM8bg2qhwePTFkI=
 github.com/edgexfoundry/go-mod-configuration/v4 v4.0.0-dev.10 h1:DMv5LZDxcqUeb1dREMd/vK+reXmZYlpafgtm8XhYdHQ=
 github.com/edgexfoundry/go-mod-configuration/v4 v4.0.0-dev.10/go.mod h1:ltUpMcOpJSzmabBtZox5qg1AK2wEikvZJyIBXtJ7mUQ=
 github.com/edgexfoundry/go-mod-core-contracts/v4 v4.0.0-dev.19 h1:uXZml7n/I/+c7k3eZRuJRrlWjYx/Euk8tlnBqeitvB8=
@@ -504,8 +504,8 @@ golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8U
 golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
 golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
 golang.org/x/crypto v0.1.0/go.mod h1:RecgLatLF4+eUMCP1PoPZQb+cVrJcOPbHkTkbkB9sbw=
-golang.org/x/crypto v0.32.0 h1:euUpcYgM8WcP71gNpTqQCn6rC2t6ULUPiOzfWaXVVfc=
-golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
+golang.org/x/crypto v0.31.0 h1:ihbySMvVjLAeSH1IbfcRTkD/iNscyz8rGzjF/E5hV6U=
+golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
 golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -585,8 +585,8 @@ golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLd
 golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
 golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
 golang.org/x/net v0.1.0/go.mod h1:Cx3nUiGt4eDBEyega/BKRp+/AlGL8hYe7U9odMt2Cco=
-golang.org/x/net v0.34.0 h1:Mb7Mrk043xzHgnRM88suvJFwzVrRfHEHJEl5/71CKw0=
-golang.org/x/net v0.34.0/go.mod h1:di0qlW3YNM5oh6GqDGQr92MyTozJPmybPK4Ev/Gm31k=
+golang.org/x/net v0.33.0 h1:74SYHlV8BIgHIFC/LrYkOGIwL19eTYXQ5wc6TBuO36I=
+golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -674,13 +674,13 @@ golang.org/x/sys v0.1.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.8.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.11.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.29.0 h1:TPYlXGxvx1MGTn2GiZDhnjPA9wZzZeGKHHmKhHYvgaU=
-golang.org/x/sys v0.29.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/sys v0.28.0 h1:Fksou7UEQUWlKvIdsqzJmUmCX3cZuD2+P3XyyzwMhlA=
+golang.org/x/sys v0.28.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.1.0/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
-golang.org/x/term v0.28.0 h1:/Ts8HFuMR2E6IP/jlo7QVLZHggjKQbhu/7H0LJFr3Gg=
-golang.org/x/term v0.28.0/go.mod h1:Sw/lC2IAUZ92udQNf3WodGtn4k/XoLyZoh8v/8uiwek=
+golang.org/x/term v0.27.0 h1:WP60Sv1nlK1T6SupCHbXzSaN0b9wUmsPoRS9b61A23Q=
+golang.org/x/term v0.27.0/go.mod h1:iMsnZpn0cago0GOrHO2+Y7u7JPn5AylBrcoWkElMTSM=
 golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
diff --git a/internal/application/callback.go b/internal/application/callback.go
index c53cd477..eebf49c6 100644
--- a/internal/application/callback.go
+++ b/internal/application/callback.go
@@ -1,6 +1,6 @@
 // -*- Mode: Go; indent-tabs-mode: t -*-
 //
-// Copyright (C) 2020-2023 IOTech Ltd
+// Copyright (C) 2020-2025 IOTech Ltd
 //
 // SPDX-License-Identifier: Apache-2.0
 
@@ -96,6 +96,10 @@ func AddDevice(addDeviceRequest requests.AddDeviceRequest, dic *di.Container) er
 		return errors.NewCommonEdgeX(errors.KindServerError, errMsg, err)
 	}
 
+	config := container.ConfigurationFrom(dic.Get)
+	reqFailsTracker := container.AllowedRequestFailuresTrackerFrom(dic.Get)
+	reqFailsTracker.Set(device.Name, int(config.Device.AllowedFails))
+
 	lc.Debugf("starting AutoEvents for device %s", device.Name)
 	container.AutoEventManagerFrom(dic.Get).RestartForDevice(device.Name)
 	return nil
@@ -190,6 +194,9 @@ func DeleteDevice(name string, dic *di.Container) errors.EdgeX {
 		return errors.NewCommonEdgeX(errors.KindServerError, errMsg, err)
 	}
 
+	reqFailsTracker := container.AllowedRequestFailuresTrackerFrom(dic.Get)
+	reqFailsTracker.Remove(device.Name)
+
 	return nil
 }
 
diff --git a/internal/application/command.go b/internal/application/command.go
index 5752e800..87d14c64 100644
--- a/internal/application/command.go
+++ b/internal/application/command.go
@@ -1,6 +1,6 @@
 // -*- Mode: Go; indent-tabs-mode: t -*-
 //
-// Copyright (C) 2020-2023 IOTech Ltd
+// Copyright (C) 2020-2025 IOTech Ltd
 //
 // SPDX-License-Identifier: Apache-2.0
 
@@ -34,20 +34,27 @@ import (
 	"github.com/edgexfoundry/go-mod-core-contracts/v4/models"
 )
 
-func GetCommand(ctx context.Context, deviceName string, commandName string, queryParams string, regexCmd bool, dic *di.Container) (*dtos.Event, errors.EdgeX) {
+func GetCommand(ctx context.Context, deviceName string, commandName string, queryParams string, regexCmd bool, dic *di.Container) (res *dtos.Event, err errors.EdgeX) {
 	if deviceName == "" {
 		return nil, errors.NewCommonEdgeX(errors.KindContractInvalid, "device name is empty", nil)
 	}
 	if commandName == "" {
 		return nil, errors.NewCommonEdgeX(errors.KindContractInvalid, "command is empty", nil)
 	}
+	var device models.Device
+	defer func() {
+		if err != nil {
+			DeviceRequestFailed(deviceName, dic)
+		} else {
+			DeviceRequestSucceeded(device, dic)
+		}
+	}()
 
-	device, err := validateServiceAndDeviceState(deviceName, dic)
+	device, err = validateServiceAndDeviceState(deviceName, dic)
 	if err != nil {
 		return nil, errors.NewCommonEdgeXWrapper(err)
 	}
 
-	var res *dtos.Event
 	_, cmdExist := cache.Profiles().DeviceCommand(device.ProfileName, commandName)
 	if cmdExist {
 		res, err = readDeviceCommand(device, commandName, queryParams, dic)
@@ -68,20 +75,27 @@ func GetCommand(ctx context.Context, deviceName string, commandName string, quer
 	return res, nil
 }
 
-func SetCommand(ctx context.Context, deviceName string, commandName string, queryParams string, requests map[string]any, dic *di.Container) (*dtos.Event, errors.EdgeX) {
+func SetCommand(ctx context.Context, deviceName string, commandName string, queryParams string, requests map[string]any, dic *di.Container) (event *dtos.Event, err errors.EdgeX) {
 	if deviceName == "" {
 		return nil, errors.NewCommonEdgeX(errors.KindContractInvalid, "device name is empty", nil)
 	}
 	if commandName == "" {
 		return nil, errors.NewCommonEdgeX(errors.KindContractInvalid, "command is empty", nil)
 	}
+	var device models.Device
+	defer func() {
+		if err != nil {
+			DeviceRequestFailed(deviceName, dic)
+		} else {
+			DeviceRequestSucceeded(device, dic)
+		}
+	}()
 
-	device, err := validateServiceAndDeviceState(deviceName, dic)
+	device, err = validateServiceAndDeviceState(deviceName, dic)
 	if err != nil {
 		return nil, errors.NewCommonEdgeXWrapper(err)
 	}
 
-	var event *dtos.Event
 	_, cmdExist := cache.Profiles().DeviceCommand(device.ProfileName, commandName)
 	if cmdExist {
 		event, err = writeDeviceCommand(device, commandName, queryParams, requests, dic)
@@ -447,8 +461,17 @@ func validateServiceAndDeviceState(deviceName string, dic *di.Container) (models
 		return models.Device{}, errors.NewCommonEdgeX(errors.KindServiceLocked, fmt.Sprintf("device %s locked", device.Name), nil)
 	}
 	// check device's OperatingState
+	// if it's a device return attempt, operating state is allowed to be DOWN
 	if device.OperatingState == models.Down {
-		return models.Device{}, errors.NewCommonEdgeX(errors.KindServiceLocked, fmt.Sprintf("device %s OperatingState is DOWN", device.Name), nil)
+		err := errors.NewCommonEdgeX(errors.KindServiceLocked, fmt.Sprintf("device %s OperatingState is DOWN", device.Name), nil)
+		config := container.ConfigurationFrom(dic.Get)
+		if config.Device.AllowedFails == 0 || config.Device.DeviceDownTimeout == 0 {
+			return models.Device{}, err
+		}
+		reqFailsTracker := container.AllowedRequestFailuresTrackerFrom(dic.Get)
+		if reqFailsTracker.Value(deviceName) > 0 {
+			return models.Device{}, err
+		}
 	}
 
 	// check device's ProfileName
diff --git a/internal/application/devicereturn.go b/internal/application/devicereturn.go
new file mode 100644
index 00000000..3befdb8a
--- /dev/null
+++ b/internal/application/devicereturn.go
@@ -0,0 +1,108 @@
+// -*- Mode: Go; indent-tabs-mode: t -*-
+//
+// Copyright (C) 2025 IOTech Ltd
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package application
+
+import (
+	"context"
+	"time"
+
+	bootstrapContainer "github.com/edgexfoundry/go-mod-bootstrap/v4/bootstrap/container"
+	"github.com/edgexfoundry/go-mod-bootstrap/v4/di"
+
+	"github.com/edgexfoundry/go-mod-core-contracts/v4/common"
+	"github.com/edgexfoundry/go-mod-core-contracts/v4/models"
+
+	"github.com/edgexfoundry/device-sdk-go/v4/internal/cache"
+	sdkCommon "github.com/edgexfoundry/device-sdk-go/v4/internal/common"
+	"github.com/edgexfoundry/device-sdk-go/v4/internal/container"
+)
+
+func deviceReturn(deviceName string, dic *di.Container) {
+	lc := bootstrapContainer.LoggingClientFrom(dic.Get)
+	dc := bootstrapContainer.DeviceClientFrom(dic.Get)
+	config := container.ConfigurationFrom(dic.Get)
+
+	for {
+	LOOP:
+		time.Sleep(time.Duration(config.Device.DeviceDownTimeout) * time.Second)
+		lc.Infof("Checking operational state for device: %s", deviceName)
+
+		d, found := cache.Devices().ForName(deviceName)
+		if !found {
+			lc.Warnf("Device %s not found. Exiting retry loop.", deviceName)
+			return
+		}
+
+		if d.OperatingState == models.Up {
+			lc.Infof("Device %s is already operational. Exiting retry loop.", deviceName)
+			return
+		}
+
+		p, found := cache.Profiles().ForName(d.ProfileName)
+		if !found {
+			lc.Warnf("Device %s has no profile. Cannot set operational state automatically.", deviceName)
+			return
+		}
+
+		for _, dr := range p.DeviceResources {
+			if dr.Properties.ReadWrite == common.ReadWrite_R ||
+				dr.Properties.ReadWrite == common.ReadWrite_RW ||
+				dr.Properties.ReadWrite == common.ReadWrite_WR {
+				_, err := GetCommand(context.Background(), deviceName, dr.Name, "", true, dic)
+				if err == nil {
+					lc.Infof("Device %s responsive: setting operational state to up.", deviceName)
+					sdkCommon.UpdateOperatingState(deviceName, models.Up, lc, dc)
+					return
+				} else {
+					lc.Errorf("Device %s unresponsive: retrying in %v seconds.", deviceName, config.Device.DeviceDownTimeout)
+					goto LOOP
+				}
+			}
+		}
+		lc.Infof("Device %s has no readable resources. Setting operational state to up without checking.", deviceName)
+		sdkCommon.UpdateOperatingState(deviceName, models.Up, lc, dc)
+		return
+	}
+}
+
+func DeviceRequestFailed(deviceName string, dic *di.Container) {
+	config := container.ConfigurationFrom(dic.Get)
+	if config.Device.AllowedFails > 0 {
+		lc := bootstrapContainer.LoggingClientFrom(dic.Get)
+		dc := bootstrapContainer.DeviceClientFrom(dic.Get)
+		reqFailsTracker := container.AllowedRequestFailuresTrackerFrom(dic.Get)
+
+		if reqFailsTracker.Decrease(deviceName) == 0 {
+			d, ok := cache.Devices().ForName(deviceName)
+			if !ok {
+				return
+			}
+			if d.OperatingState != models.Down {
+				lc.Infof("Marking device %s non-operational", deviceName)
+				sdkCommon.UpdateOperatingState(deviceName, models.Down, lc, dc)
+			}
+			if config.Device.DeviceDownTimeout > 0 {
+				lc.Warnf("Will retry device %s in %v seconds", deviceName, config.Device.DeviceDownTimeout)
+				go deviceReturn(deviceName, dic)
+			}
+			return
+		}
+	}
+}
+
+func DeviceRequestSucceeded(d models.Device, dic *di.Container) {
+	config := container.ConfigurationFrom(dic.Get)
+	reqFailsTracker := container.AllowedRequestFailuresTrackerFrom(dic.Get)
+	if config.Device.AllowedFails > 0 && reqFailsTracker.Value(d.Name) < int(config.Device.AllowedFails) {
+		reqFailsTracker.Set(d.Name, int(config.Device.AllowedFails))
+		if d.OperatingState == models.Down {
+			lc := bootstrapContainer.LoggingClientFrom(dic.Get)
+			dc := bootstrapContainer.DeviceClientFrom(dic.Get)
+			sdkCommon.UpdateOperatingState(d.Name, models.Up, lc, dc)
+		}
+	}
+}
diff --git a/internal/autoevent/executor.go b/internal/autoevent/executor.go
index 473add0a..6d7a195e 100644
--- a/internal/autoevent/executor.go
+++ b/internal/autoevent/executor.go
@@ -1,6 +1,6 @@
 // -*- Mode: Go; indent-tabs-mode: t -*-
 //
-// Copyright (C) 2019-2023 IOTech Ltd
+// Copyright (C) 2019-2025 IOTech Ltd
 //
 // SPDX-License-Identifier: Apache-2.0
 
diff --git a/internal/cache/devices.go b/internal/cache/devices.go
index b92adcfc..a6088635 100644
--- a/internal/cache/devices.go
+++ b/internal/cache/devices.go
@@ -1,6 +1,6 @@
 // -*- Mode: Go; indent-tabs-mode: t -*-
 //
-// Copyright (C) 2020-2023 IOTech Ltd
+// Copyright (C) 2020-2025 IOTech Ltd
 //
 // SPDX-License-Identifier: Apache-2.0
 
@@ -50,13 +50,10 @@ type deviceCache struct {
 func newDeviceCache(devices []models.Device, dic *di.Container) DeviceCache {
 	defaultSize := len(devices)
 	dMap := make(map[string]*models.Device, defaultSize)
-	for i, d := range devices {
-		dMap[d.Name] = &devices[i]
-	}
-
 	dc = &deviceCache{deviceMap: dMap, dic: dic}
 	lastConnectedMetrics := make(map[string]gometrics.Gauge)
 	for _, d := range devices {
+		dMap[d.Name] = &d
 		deviceMetric := gometrics.NewGauge()
 		registerMetric(d.Name, deviceMetric, dic)
 		lastConnectedMetrics[d.Name] = deviceMetric
diff --git a/internal/config/types.go b/internal/config/types.go
index b6437f81..e05adaea 100644
--- a/internal/config/types.go
+++ b/internal/config/types.go
@@ -1,7 +1,7 @@
 // -*- mode: Go; indent-tabs-mode: t -*-
 //
 // Copyright (C) 2017-2018 Canonical Ltd
-// Copyright (C) 2018-2023 IOTech Ltd
+// Copyright (C) 2018-2025 IOTech Ltd
 // Copyright (c) 2021 Intel Corporation
 //
 // SPDX-License-Identifier: Apache-2.0
@@ -53,6 +53,10 @@ type DeviceInfo struct {
 	EnableAsyncReadings bool
 	// Labels are properties applied to the device service to help with searching
 	Labels []string
+	// AllowedFails specifies the number of failed requests allowed before a device is marked as down.
+	AllowedFails uint
+	// DeviceDownTimeout specifies the duration in seconds that the Device Service will try to contact a device if it is marked as down.
+	DeviceDownTimeout uint
 }
 
 // DiscoveryInfo is a struct which contains configuration of device auto discovery.
diff --git a/internal/container/allowedfailstracker.go b/internal/container/allowedfailstracker.go
new file mode 100644
index 00000000..b05fd7c9
--- /dev/null
+++ b/internal/container/allowedfailstracker.go
@@ -0,0 +1,59 @@
+// -*- Mode: Go; indent-tabs-mode: t -*-
+//
+// Copyright (C) 2025 IOTech Ltd
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package container
+
+// AllowedFailuresTracker wraps a map of device names to atomic integers that track the number of allowed request
+// failures for each device.
+type AllowedFailuresTracker struct {
+	data map[string]*AtomicInt
+}
+
+// NewAllowedFailuresTracker creates and initializes a new tracker.
+func NewAllowedFailuresTracker() AllowedFailuresTracker {
+	return AllowedFailuresTracker{
+		data: make(map[string]*AtomicInt),
+	}
+}
+
+// Get retrieves the AtomicInt for a given device name.
+// Returns nil if the device does not exist.
+func (aft *AllowedFailuresTracker) Get(deviceName string) *AtomicInt {
+	return aft.data[deviceName]
+}
+
+// Set initializes or updates the AtomicInt for a given device.
+func (aft *AllowedFailuresTracker) Set(deviceName string, value int) {
+	if _, exists := aft.data[deviceName]; !exists {
+		aft.data[deviceName] = &AtomicInt{}
+	}
+	aft.data[deviceName].Set(value)
+}
+
+// Decrease decreases the AtomicInt value for a given device by 1.
+// Returns the updated value or -1 if the device does not exist.
+func (aft *AllowedFailuresTracker) Decrease(deviceName string) int {
+	if atomicInt, exists := aft.data[deviceName]; exists {
+		if atomicInt.Value() >= 0 {
+			return atomicInt.Decrease()
+		}
+	}
+	return -1
+}
+
+// Value retrieves the current value of the AtomicInt for a device.
+// Returns -1 if the device does not exist.
+func (aft *AllowedFailuresTracker) Value(deviceName string) int {
+	if atomicInt, exists := aft.data[deviceName]; exists {
+		return atomicInt.Value()
+	}
+	return -1
+}
+
+// Remove deletes the entry for a given device name from the tracker.
+func (aft *AllowedFailuresTracker) Remove(deviceName string) {
+	delete(aft.data, deviceName)
+}
diff --git a/internal/container/atomicint.go b/internal/container/atomicint.go
new file mode 100644
index 00000000..f2cc09f3
--- /dev/null
+++ b/internal/container/atomicint.go
@@ -0,0 +1,34 @@
+// -*- Mode: Go; indent-tabs-mode: t -*-
+//
+// Copyright (C) 2025 IOTech Ltd
+//
+// SPDX-License-Identifier: Apache-2.0
+
+package container
+
+import "sync"
+
+type AtomicInt struct {
+	mutex sync.RWMutex
+	value int
+}
+
+func (i *AtomicInt) Value() int {
+	i.mutex.RLock()
+	defer i.mutex.RUnlock()
+	v := i.value
+	return v
+}
+
+func (i *AtomicInt) Decrease() int {
+	i.mutex.Lock()
+	defer i.mutex.Unlock()
+	i.value--
+	return i.value
+}
+
+func (i *AtomicInt) Set(v int) {
+	i.mutex.Lock()
+	defer i.mutex.Unlock()
+	i.value = v
+}
diff --git a/internal/container/deviceservice.go b/internal/container/deviceservice.go
index 4a3fed8e..00f4f114 100644
--- a/internal/container/deviceservice.go
+++ b/internal/container/deviceservice.go
@@ -1,6 +1,6 @@
 // -*- Mode: Go; indent-tabs-mode: t -*-
 //
-// Copyright (C) 2020-2023 IOTech Ltd
+// Copyright (C) 2020-2025 IOTech Ltd
 //
 // SPDX-License-Identifier: Apache-2.0
 
@@ -60,3 +60,11 @@ func DiscoveryRequestIdFrom(get di.Get) string {
 	}
 	return id
 }
+
+// AllowedRequestFailuresTrackerName contains the name of allowed request failures tracker in the DIC.
+var AllowedRequestFailuresTrackerName = di.TypeInstanceToName(AllowedFailuresTracker{})
+
+// AllowedRequestFailuresTrackerFrom helper function queries the DIC and returns a device request failures tracker.
+func AllowedRequestFailuresTrackerFrom(get di.Get) AllowedFailuresTracker {
+	return get(AllowedRequestFailuresTrackerName).(AllowedFailuresTracker)
+}
diff --git a/pkg/service/init.go b/pkg/service/init.go
index 502021d7..85805608 100644
--- a/pkg/service/init.go
+++ b/pkg/service/init.go
@@ -63,6 +63,18 @@ func (b *Bootstrap) BootstrapHandler(ctx context.Context, wg *sync.WaitGroup, _
 		return false
 	}
 
+	devices := cache.Devices().All()
+	config := container.ConfigurationFrom(dic.Get)
+	reqFailsTracker := container.NewAllowedFailuresTracker()
+	for _, d := range devices {
+		reqFailsTracker.Set(d.Name, int(config.Device.AllowedFails))
+	}
+	dic.Update(di.ServiceConstructorMap{
+		container.AllowedRequestFailuresTrackerName: func(get di.Get) any {
+			return reqFailsTracker
+		},
+	})
+
 	if s.AsyncReadingsEnabled() {
 		s.asyncCh = make(chan *models.AsyncValues, s.config.Device.AsyncBufferSize)
 		wg.Add(1)


Test patch:
diff --git a/internal/controller/http/command_test.go b/internal/controller/http/command_test.go
index e3e2744f..89b0b330 100644
--- a/internal/controller/http/command_test.go
+++ b/internal/controller/http/command_test.go
@@ -1,5 +1,5 @@
 //
-// Copyright (C) 2022-2023 IOTech Ltd
+// Copyright (C) 2022-2025 IOTech Ltd
 //
 // SPDX-License-Identifier: Apache-2.0
 
@@ -208,6 +208,9 @@ func mockDic() *di.Container {
 		bootstrapContainer.MetricsManagerInterfaceName: func(get di.Get) interface{} {
 			return mockMetricsManager
 		},
+		container.AllowedRequestFailuresTrackerName: func(get di.Get) any {
+			return container.NewAllowedFailuresTracker()
+		},
 	})
 
 	return dic