rtl8723ae: Fix some sparse warnings

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors and warnings in os_dep/ioctl_linux.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/os_dep/ioctl_linux.c
/home/finger/rtl8723au/os_dep/ioctl_linux.c:1632 rtw_wx_get_range() warn: assigning (-78) to unsigned variable 'range->avg_qual.level'
/home/finger/rtl8723au/os_dep/ioctl_linux.c:5122 rtw_p2p_get_invitation_procedure() warn: should this be a bitwise op?
/home/finger/rtl8723au/os_dep/ioctl_linux.c:6320 rtw_p2p_get2() warn: statement has no effect 5
/home/finger/rtl8723au/os_dep/ioctl_linux.c:6603 rtw_dbg_port() warn: unsigned 'extra_arg' is never less than zero.
/home/finger/rtl8723au/os_dep/ioctl_linux.c:7493 set_group_key() warn: missing break? reassigning 'keylen'
/home/finger/rtl8723au/os_dep/ioctl_linux.c:9936:24: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10115:24: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10233:35: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10265:35: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10314:35: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10340:35: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10380:35: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10565:28: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10612:35: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10731:35: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10929:27: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10953:33: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:10982:33: error: bad constant expression
/home/finger/rtl8723au/os_dep/ioctl_linux.c:11023:35: error: bad constant expression

Several of these are real errors in the code.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors/warnings in os_dep/osdep_service.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/os_dep/osdep_service.c
/home/finger/rtl8723au/os_dep/osdep_service.c:1275:19: warning: non-ANSI function declaration of function 'rtw_yield_os'
/home/finger/rtl8723au/os_dep/osdep_service.c:1298:35: warning: non-ANSI function declaration of function 'rtw_suspend_lock_init'
/home/finger/rtl8723au/os_dep/osdep_service.c:1307:37: warning: non-ANSI function declaration of function 'rtw_suspend_lock_uninit'
/home/finger/rtl8723au/os_dep/osdep_service.c:1316:30: warning: non-ANSI function declaration of function 'rtw_lock_suspend'
/home/finger/rtl8723au/os_dep/osdep_service.c:1329:32: warning: non-ANSI function declaration of function 'rtw_unlock_suspend'
/home/finger/rtl8723au/os_dep/osdep_service.c:1775 rtw_change_ifname() warn: variable dereferenced before check 'padapter' (see line 1771)

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch warning in core/rtw_ieee80211.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_ieee80211.c
/home/finger/rtl8723au/core/rtw_ieee80211.c:167 rtw_set_ie() info: ignoring unreachable code.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fis smatch errors in core/rtw_cmd.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_cmd.c
/home/finger/rtl8723au/core/rtw_cmd.c:500 rtw_cmd_thread() error: buffer overflow 'wlancmds' 63 <= 63
/home/finger/rtl8723au/core/rtw_cmd.c:522 rtw_cmd_thread() error: buffer overflow 'rtw_cmd_callback' 63 <= 63

Both of these are off-by-one errors that could lead to trying to
access memory outside the boundaries of the array specified.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch warnings in core/rtw_security.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_security.c
/home/finger/rtl8723au/core/rtw_security.c:2486 rijndaelKeySetupEnc() warn: '4' 4 can't fit into 4611686018427387903 'rk'
/home/finger/rtl8723au/core/rtw_security.c:2533 rijndaelEncrypt() warn: '8' 8 can't fit into 4611686018427387903 'rk'

These are both smatch errors, not problems with the code.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch warning in core/rtw_debug.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_debug.c
/home/finger/rtl8723au/core/rtw_debug.c:688 proc_set_rx_signal() warn: unsigned 'signal_strength' is never less than zero.

The offending statement is testing an unsigned variable.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors/warnings in core/rtw_ioctl_set.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_ioctl_set.c
/home/finger/rtl8723au/core/rtw_ioctl_set.c:822 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:824 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKeylen' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:828 rtw_set_802_11_add_wep() error: buffer overflow 'psecuritypriv->dot11DefKey' 4 <= 4
/home/finger/rtl8723au/core/rtw_ioctl_set.c:976 rtw_set_802_11_add_key() warn: was && intended here instead of ||?

The last of these is a logical error. The others are due to an off-by-one error.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch warning in os_dep/usb_intf.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/os_dep/usb_intf.c
/home/finger/rtl8723au/os_dep/usb_intf.c:1005 rtw_resume_process() error: potentially dereferencing uninitialized 'pwrpriv'.

The fix is to add the necessary test.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors/warnings in hal/rtl8723a_bt-coexist.c

Smatch shows the following:
  CHECK   /home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:612 bthci_GetAssocInfo() warn: should this be a bitwise op?
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:743 bthci_CheckLogLinkBehavior() warn: should this be a bitwise op?
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:4065 bthci_CmdLogicalLinkCancel() error: buffer overflow 'pBTinfo->BtAsocEntry[CurrentEntryNum]->LogLinkCmdData' 2 <= 255
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:4066 bthci_CmdLogicalLinkCancel() error: buffer overflow 'pBTinfo->BtAsocEntry[CurrentEntryNum]->LogLinkCmdData' 2 <= 255
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:4073 bthci_CmdLogicalLinkCancel() error: buffer overflow 'pBTinfo->BtAsocEntry[CurrentEntryNum]->LogLinkCmdData' 2 <= 255
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:6463 BTHCI_EventParse() info: ignoring unreachable code.
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:6463 BTHCI_EventParse() info: ignoring unreachable code.
/home/finger/rtl8723au/hal/rtl8723a_bt-coexist.c:8528 BTDM_1AntLpsLeave() warn: statement has no effect 22

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors/warnings for hal/HalDMOutSrc8192C_CE.c

Smatch shows the following:
  CHECK   /home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:313 odm_TXPowerTrackingCallback_ThermalMeter_92C() error: buffer overflow 'TxPwrLevel' 2 <= 2
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:313 odm_TXPowerTrackingCallback_ThermalMeter_92C() error: buffer overflow 'TxPwrLevel' 2 <= 2
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:327 odm_TXPowerTrackingCallback_ThermalMeter_92C() error: buffer overflow 'TxPwrLevel' 2 <= 2
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:327 odm_TXPowerTrackingCallback_ThermalMeter_92C() error: buffer overflow 'TxPwrLevel' 2 <= 2
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:334 odm_TXPowerTrackingCallback_ThermalMeter_92C() error: buffer overflow 'TxPwrLevel' 2 <= 2
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:334 odm_TXPowerTrackingCallback_ThermalMeter_92C() error: buffer overflow 'TxPwrLevel' 2 <= 2
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:2014 rtl8192c_PHY_APCalibrate() info: ignoring unreachable code.
/home/finger/rtl8723au/hal/HalDMOutSrc8192C_CE.c:2014 rtl8192c_PHY_APCalibrate() info: ignoring unreachable code.

The buffer overflow errors are due to an off-by-one error.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch warnings in   CHECK   /home/finger/rtl8723au/hal/odm_HWConfig.c

Smatch lists the following:
/home/finger/rtl8723au/hal/odm_HWConfig.c:400 odm_RxPhyStatus92CSeries_Parsing() warn: always true condition '(pPktinfo->Rate >= 0) => (0-255 >= 0)'
/home/finger/rtl8723au/hal/odm_HWConfig.c:783 odm_Process_RSSIForDM() warn: always true condition '(pPktinfo->Rate >= 0) => (0-255 >= 0)'

Variable Rate is unsigned and always >= 0.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch problems in hal/rtl8723a_mp.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/hal/rtl8723a_mp.c
/home/finger/rtl8723au/hal/rtl8723a_mp.c:538 Hal_SetAntennaPathPower() error: buffer overflow 'TxPowerLevel' 2 <= 2
/home/finger/rtl8723au/hal/rtl8723a_mp.c:581 Hal_SetTxPower() error: buffer overflow 'TxPowerLevel' 2 <= 2
/home/finger/rtl8723au/hal/rtl8723a_mp.c:599 Hal_SetTxAGCOffset() info: ignoring unreachable code.
/home/finger/rtl8723au/hal/rtl8723a_mp.c:599 Hal_SetTxAGCOffset() info: ignoring unreachable code.

The buffer overflow errors are again an off-by=one situation.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch problems in hal/usb_ops_linux.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/hal/usb_ops_linux.c
/home/finger/rtl8723au/hal/usb_ops_linux.c:1723:1: warning: no newline at end of file
/home/finger/rtl8723au/hal/usb_ops_linux.c:1579 usb_read_port() warn: variable dereferenced before check 'precvbuf' (see line 1573)

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors in hal/rtl8723a_rf6052.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/hal/rtl8723a_rf6052.c
/home/finger/rtl8723au/hal/rtl8723a_rf6052.c:973 PHY_RFShadowRefresh() error: buffer overflow 'RF_Shadow[eRFPath]' 63 <= 63
/home/finger/rtl8723au/hal/rtl8723a_rf6052.c:974 PHY_RFShadowRefresh() error: buffer overflow 'RF_Shadow[eRFPath]' 63 <= 63
/home/finger/rtl8723au/hal/rtl8723a_rf6052.c:975 PHY_RFShadowRefresh() error: buffer overflow 'RF_Shadow[eRFPath]' 63 <= 63
/home/finger/rtl8723au/hal/rtl8723a_rf6052.c:976 PHY_RFShadowRefresh() error: buffer overflow 'RF_Shadow[eRFPath]' 63 <= 63
/home/finger/rtl8723au/hal/rtl8723a_rf6052.c:977 PHY_RFShadowRefresh() error: buffer overflow 'RF_Shadow[eRFPath]' 63 <= 63

These are all due to an off-by-one error in a for loop.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch warnings in hal/usb_halinit.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/hal/usb_halinit.c
/home/finger/rtl8723au/hal/usb_halinit.c:1402 HwSuspendModeEnable92Cu() info: ignoring unreachable code.
/home/finger/rtl8723au/hal/usb_halinit.c:1402 HwSuspendModeEnable92Cu() info: ignoring unreachable code.
/home/finger/rtl8723au/hal/usb_halinit.c:2931 _ReadIDs() warn: was || intended here instead of &&?

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors/warnings in core/rtw_xmit.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_xmit.c
/home/finger/rtl8723au/core/rtw_xmit.c:2454 rtw_free_xmitframe() error: potentially dereferencing uninitialized 'queue'.
/home/finger/rtl8723au/core/rtw_xmit.c:2558 dequeue_one_xmitframe() info: ignoring unreachable code.
/home/finger/rtl8723au/core/rtw_xmit.c:2558 dequeue_one_xmitframe() info: ignoring unreachable code.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors in core/rtw_recv.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_recv.c
/home/finger/rtl8723au/core/rtw_recv.c:2865 amsdu_to_msdu() warn: variable dereferenced before check 'sub_skb' (see line 2843)

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors in core/rtw_wlan_util.c

Smatch lists the following:
  CHECK   /home/finger/rtl8723au/core/rtw_wlan_util.c
/home/finger/rtl8723au/core/rtw_wlan_util.c:1646 rtw_check_bcn_info() info: ignoring unreachable code.
/home/finger/rtl8723au/core/rtw_wlan_util.c:1646 rtw_check_bcn_info() info: ignoring unreachable code.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>
rtl8723au: Fix smatch errors and warnings in core/rtw_mlme_ext.c

Smatch reports the following:
  CHECK   /home/finger/rtl8723au/core/rtw_mlme_ext.c
/home/finger/rtl8723au/core/rtw_mlme_ext.c:2053 OnAssocReq() debug: remove_pools: nr_children over 4000 (4200). (reassoc 0-1)
/home/finger/rtl8723au/core/rtw_mlme_ext.c:2053 OnAssocReq() error: buffer overflow 'pstapriv->sta_aid' 32 <= 32
/home/finger/rtl8723au/core/rtw_mlme_ext.c:5473 on_action_public_p2p() warn: if();
/home/finger/rtl8723au/core/rtw_mlme_ext.c:9690 process_80211d() error: testing array offset 'i' after use.

Of these, the last one appears to be a smatch error as I find nothing wrong.

Signed-off-by: Larry Finger <Larry.Finger@lwfinger.net>