25.19. Modifying Link Loss Behavior This section describes how to modify the link loss behavior of devices that use either Fibre Channel or iSCSI protocols. 25.19.1. Fibre Channel If a driver implements the Transport dev_loss_tmo callback, access attempts to a device through a link will be blocked when a transport problem is detected. To verify if a device is blocked, run the following command: This command will return blocked if the device is blocked. If the device is operating normally, this command will return running . Procedure 25.15. Determining the State of a Remote Port To determine the state of a remote port, run the following command: This command will return Blocked when the remote port (along with devices accessed through it) are blocked. If the remote port is operating normally, the command will return Online . If the problem is not resolved within dev_loss_tmo seconds, the rport and devices will be unblocked and all I/O running on that device (along with any new I/O sent to that device) will be failed. Procedure 25.16. Changing dev_loss_tmo To change the dev_loss_tmo value, echo in the desired value to the file. For example, to set dev_loss_tmo to 30 seconds, run: For more information about dev_loss_tmo , refer to Section 25.4.1, "Fibre Channel API" . When a link loss exceeds dev_loss_tmo , the scsi_device and sd N devices are removed. Typically, the Fibre Channel class will leave the device as is; i.e. /dev/sd x will remain /dev/sd x . This is because the target binding is saved by the Fibre Channel driver so when the target port returns, the SCSI addresses are recreated faithfully. However, this cannot be guaranteed; the sd x will be restored only if no additional change on in-storage box configuration of LUNs is made. 25.19.2. iSCSI Settings with dm-multipath If dm-multipath is implemented, it is advisable to set iSCSI timers to immediately defer commands to the multipath layer. To configure this, nest the following line under device { in /etc/multipath.conf : This ensures that I/O errors are retried and queued if all paths are failed in the dm-multipath layer. You may need to adjust iSCSI timers further to better monitor your SAN for problems. Available iSCSI timers you can configure are NOP-Out Interval/Timeouts and replacement_timeout , which are discussed in the following sections. 25.19.2.1. NOP-Out Interval/Timeout To help monitor problems the SAN, the iSCSI layer sends a NOP-Out request to each target. If a NOP-Out request times out, the iSCSI layer responds by failing any running commands and instructing the SCSI layer to requeue those commands when possible. When dm-multipath is being used, the SCSI layer will fail those running commands and defer them to the multipath layer. The multipath layer then retries those commands on another path. If dm-multipath is not being used, those commands are retried five times before failing altogether. Intervals between NOP-Out requests are 5 seconds by default. To adjust this, open /etc/iscsi/iscsid.conf and edit the following line: Once set, the iSCSI layer will send a NOP-Out request to each target every [interval value] seconds. By default, NOP-Out requests time out in 5 seconds [9] . To adjust this, open /etc/iscsi/iscsid.conf and edit the following line: This sets the iSCSI layer to timeout a NOP-Out request after [timeout value] seconds. SCSI Error Handler If the SCSI Error Handler is running, running commands on a path will not be failed immediately when a NOP-Out request times out on that path. Instead, those commands will be failed after replacement_timeout seconds. For more information about replacement_timeout , refer to Section 25.19.2.2, " replacement_timeout " . To verify if the SCSI Error Handler is running, run: 25.19.2.2. replacement_timeout replacement_timeout controls how long the iSCSI layer should wait for a timed-out path/session to reestablish itself before failing any commands on it. The default replacement_timeout value is 120 seconds. To adjust replacement_timeout , open /etc/iscsi/iscsid.conf and edit the following line: The 1 queue_if_no_path option in /etc/multipath.conf sets iSCSI timers to immediately defer commands to the multipath layer (refer to Section 25.19.2, "iSCSI Settings with dm-multipath " ). This setting prevents I/O errors from propagating to the application; because of this, you can set replacement_timeout to 15-20 seconds. By configuring a lower replacement_timeout , I/O is quickly sent to a new path and executed (in the event of a NOP-Out timeout) while the iSCSI layer attempts to re-establish the failed path/session. If all paths time out, then the multipath and device mapper layer will internally queue I/O based on the settings in /etc/multipath.conf instead of /etc/iscsi/iscsid.conf . Important Whether your considerations are failover speed or security, the recommended value for replacement_timeout will depend on other factors. These factors include the network, target, and system workload. As such, it is recommended that you thoroughly test any new configurations to replacements_timeout before applying it to a mission-critical system. iSCSI and DM Multipath overrides The recovery_tmo sysfs option controls the timeout for a particular iSCSI device. The following options globally override recovery_tmo values: The replacement_timeout configuration option globally overrides the recovery_tmo value for all iSCSI devices. For all iSCSI devices that are managed by DM Multipath, the fast_io_fail_tmo option in DM Multipath globally overrides the recovery_tmo value. The fast_io_fail_tmo option in DM Multipath also overrides the fast_io_fail_tmo option in Fibre Channel devices. The DM Multipath fast_io_fail_tmo option takes precedence over replacement_timeout . Red Hat does not recommend using replacement_timeout to override recovery_tmo in devices managed by DM Multipath because DM Multipath always resets recovery_tmo when the multipathd service reloads. 25.19.3. iSCSI Root When accessing the root partition directly through an iSCSI disk, the iSCSI timers should be set so that iSCSI layer has several chances to try to reestablish a path/session. In addition, commands should not be quickly re-queued to the SCSI layer. This is the opposite of what should be done when dm-multipath is implemented. To start with, NOP-Outs should be disabled. You can do this by setting both NOP-Out interval and timeout to zero. To set this, open /etc/iscsi/iscsid.conf and edit as follows: In line with this, replacement_timeout should be set to a high number. This will instruct the system to wait a long time for a path/session to reestablish itself. To adjust replacement_timeout , open /etc/iscsi/iscsid.conf and edit the following line: After configuring /etc/iscsi/iscsid.conf , you must perform a re-discovery of the affected storage. This will allow the system to load and use any new values in /etc/iscsi/iscsid.conf . For more information on how to discover iSCSI devices, refer to Section 25.15, "Scanning iSCSI Interconnects" . Configuring Timeouts for a Specific Session You can also configure timeouts for a specific session and make them non-persistent (instead of using /etc/iscsi/iscsid.conf ). To do so, run the following command (replace the variables accordingly): Important The configuration described here is recommended for iSCSI sessions involving root partition access. For iSCSI sessions involving access to other types of storage (namely, in systems that use dm-multipath ), refer to Section 25.19.2, "iSCSI Settings with dm-multipath " . [9] Prior to Red Hat Enterprise Linux 5.4, the default NOP-Out requests time out was 15 seconds.

Commands:
cat /sys/block/ device /device/state
cat /sys/class/fc_remote_port/rport- H : B : R /port_state
echo 30 > /sys/class/fc_remote_port/rport- H : B : R /dev_loss_tmo
features "1 queue_if_no_path"
node.conn[0].timeo.noop_out_interval = [interval value]
node.conn[0].timeo.noop_out_timeout = [timeout value]
iscsiadm -m session -P 3
node.session.timeo.replacement_timeout = [replacement_timeout]
node.conn[0].timeo.noop_out_interval = 0 node.conn[0].timeo.noop_out_timeout = 0
node.session.timeo.replacement_timeout = replacement_timeout
iscsiadm -m node -T target_name -p target_IP : port -o update -n node.session.timeo.replacement_timeout -v USD timeout_value