Repository: eshaham/israeli-bank-scrapers
Pull request #903: fix(beinleumi): support new UI
Author: login: Nic3Guy
Labels: name: released
Created: 2025-03-01T20:01:39Z | Merged: 2025-03-09T05:41:52Z

Fix #901 

--------------

The BeinLeumi have new UI, the transactions page is wrap with iFrame now
needed to adjust the `base-beinleumi-group.ts` 

I don't know if I did it with the best practices of Node.js, but it works, and i also ran the test and all pass (see screenshot)
![Screenshot 2025-03-01 at 21 57 20](https://github.com/user-attachments/assets/dc4f6f3c-9f41-444a-8729-67983c65a44e)


Linked issue #901: Beinleumi getting timeout after login - probably due to the new UI
When trying to scrape "BeinLeumi" account, after the login (which is successful) it pauses and get timeout with out scraping

### the error message: 
> GENERIC - Waiting for selector `#matafLogoutLink` failed: Waiting failed: 30000ms exceeded
Waiting for the debugger to disconnect...

### Additional info maybe it will help:
1. There is an overlay that suggest walkthough of the new UI - but even if manually cancelling it doesn't help, also it always return even if you tell it not to.
2. I wasn't able to locate the `#matafLogoutLink` when I search with devTools

### Screenshot
the overlay:
<img width="1199" alt="Image" src="https://github.com/user-attachments/assets/70cef2bb-cee0-4e6d-b074-4e56b00d49d5" />

Linked issue comments: user: login: baruchiro
created_at: 2025-02-23T12:42:55Z
body: Someone with a BeinLeumi account should find a way to revive the scraper...

@whatuserever
author_association: COLLABORATOR
user: login: github-actions[bot]
created_at: 2025-03-09T05:43:08Z
body: :tada: This issue has been resolved in version 5.4.1 :tada:

The release is available on [GitHub release](https://github.com/eshaham/israeli-bank-scrapers/releases/tag/v5.4.1)

Your **[semantic-release](https://github.com/semantic-release/semantic-release)** bot :package::rocket:
author_association: NONE

Diff:
diff --git a/src/scrapers/base-beinleumi-group.ts b/src/scrapers/base-beinleumi-group.ts
index b17116d0f..17be8e5eb 100644
--- a/src/scrapers/base-beinleumi-group.ts
+++ b/src/scrapers/base-beinleumi-group.ts
@@ -1,5 +1,5 @@
 import moment, { type Moment } from 'moment';
-import { type Page } from 'puppeteer';
+import { type Frame, type Page } from 'puppeteer';
 import { SHEKEL_CURRENCY, SHEKEL_CURRENCY_SYMBOL } from '../constants';
 import {
   clickButton,
@@ -25,7 +25,7 @@ const DESCRIPTION_COLUMN_CLASS_COMPLETED = 'reference wrap_normal';
 const DESCRIPTION_COLUMN_CLASS_PENDING = 'details wrap_normal';
 const REFERENCE_COLUMN_CLASS = 'details';
 const DEBIT_COLUMN_CLASS = 'debit';
-const CREDIT_COLUMN_CLASS = 'credit'; 
+const CREDIT_COLUMN_CLASS = 'credit';
 const ERROR_MESSAGE_CLASS = 'NO_DATA';
 const ACCOUNTS_NUMBER = 'div.fibi_account span.acc_num';
 const CLOSE_SEARCH_BY_DATES_BUTTON_CLASS = 'ui-datepicker-close';
@@ -51,7 +51,7 @@ interface ScrapedTransaction {
 
 export function getPossibleLoginResults(): PossibleLoginResults {
   const urls: PossibleLoginResults = {};
-  urls[LoginResults.Success] = [/FibiMenu\/Online/];
+  urls[LoginResults.Success] = [/fibi.*accountSummary/];
   urls[LoginResults.InvalidPassword] = [/FibiMenu\/Marketing\/Private\/Home/];
   return urls;
 }
@@ -134,7 +134,7 @@ function extractTransactionDetails(txnRow: TransactionsTr, transactionStatus: Tr
   return item;
 }
 
-async function getTransactionsColsTypeClasses(page: Page, tableLocator: string): Promise<TransactionsColsTypes> {
+async function getTransactionsColsTypeClasses(page: Page | Frame, tableLocator: string): Promise<TransactionsColsTypes> {
   const result: TransactionsColsTypes = {};
   const typeClassesObjs = await pageEvalAll(page, `${tableLocator} tbody tr:first-of-type td`, null, (tds) => {
     return tds.map((td, index) => ({
@@ -158,7 +158,7 @@ function extractTransaction(txns: ScrapedTransaction[], transactionStatus: Trans
   }
 }
 
-async function extractTransactions(page: Page, tableLocator: string, transactionStatus: TransactionStatuses) {
+async function extractTransactions(page: Page | Frame, tableLocator: string, transactionStatus: TransactionStatuses) {
   const txns: ScrapedTransaction[] = [];
   const transactionsColsTypes = await getTransactionsColsTypeClasses(page, tableLocator);
 
@@ -174,7 +174,7 @@ async function extractTransactions(page: Page, tableLocator: string, transaction
   return txns;
 }
 
-async function isNoTransactionInDateRangeError(page: Page) {
+async function isNoTransactionInDateRangeError(page: Page | Frame) {
   const hasErrorInfoElement = await elementPresentOnPage(page, `.${ERROR_MESSAGE_CLASS}`);
   if (hasErrorInfoElement) {
     const errorText = await page.$eval(`.${ERROR_MESSAGE_CLASS}`, (errorElement) => {
@@ -185,7 +185,7 @@ async function isNoTransactionInDateRangeError(page: Page) {
   return false;
 }
 
-async function searchByDates(page: Page, startDate: Moment) {
+async function searchByDates(page: Page | Frame, startDate: Moment) {
   await clickButton(page, 'a#tabHeader4');
   await waitUntilElementFound(page, 'div#fibi_dates');
   await fillInput(
@@ -198,7 +198,7 @@ async function searchByDates(page: Page, startDate: Moment) {
   await waitForNavigation(page);
 }
 
-async function getAccountNumber(page: Page) {
+async function getAccountNumber(page: Page | Frame) {
   const selectedSnifAccount = await page.$eval(ACCOUNTS_NUMBER, (option) => {
     return (option as HTMLElement).innerText;
   });
@@ -206,18 +206,18 @@ async function getAccountNumber(page: Page) {
   return selectedSnifAccount.replace('/', '_').trim();
 }
 
-async function checkIfHasNextPage(page: Page) {
+async function checkIfHasNextPage(page: Page | Frame) {
   return elementPresentOnPage(page, NEXT_PAGE_LINK);
 }
 
-async function navigateToNextPage(page: Page) {
+async function navigateToNextPage(page: Page | Frame) {
   await clickButton(page, NEXT_PAGE_LINK);
   await waitForNavigation(page);
 }
 
 /* Couldn't reproduce scenario with multiple pages of pending transactions - Should support if exists such case.
    needToPaginate is false if scraping pending transactions */
-async function scrapeTransactions(page: Page, tableLocator: string, transactionStatus: TransactionStatuses, needToPaginate: boolean) {
+async function scrapeTransactions(page: Page | Frame, tableLocator: string, transactionStatus: TransactionStatuses, needToPaginate: boolean) {
   const txns = [];
   let hasNextPage = false;
 
@@ -235,7 +235,7 @@ async function scrapeTransactions(page: Page, tableLocator: string, transactionS
   return convertTransactions(txns);
 }
 
-async function getAccountTransactions(page: Page) {
+async function getAccountTransactions(page: Page | Frame) {
   await Promise.race([
     waitUntilElementFound(page, 'div[id*=\'divTable\']', false),
     waitUntilElementFound(page, `.${ERROR_MESSAGE_CLASS}`, false),
@@ -257,7 +257,7 @@ async function getAccountTransactions(page: Page) {
   return txns;
 }
 
-async function getCurrentBalance(page: Page) {
+async function getCurrentBalance(page: Page | Frame) {
   const balanceElement = await page.$(CURRENT_BALANCE);
   if (!balanceElement) {
     return undefined;
@@ -270,12 +270,12 @@ async function getCurrentBalance(page: Page) {
 
 export async function waitForPostLogin(page: Page) {
   return Promise.race([
-    waitUntilElementFound(page, '#matafLogoutLink', true),
-    waitUntilElementFound(page, '#validationMsg', true),
+    waitUntilElementFound(page, '#card-header', true),
+    waitUntilElementFound(page, '#account_num', true),
   ]);
 }
 
-async function fetchAccountData(page: Page, startDate: Moment) {
+async function fetchAccountData(page: Page | Frame, startDate: Moment) {
   await searchByDates(page, startDate);
   const accountNumber = await getAccountNumber(page);
   const balance = await getCurrentBalance(page);
@@ -298,11 +298,24 @@ async function getAccountIdsBySelector(page: Page): Promise<string[]> {
   return accountsIds;
 }
 
+async function getTransactionsFrame(page: Page): Promise<Frame> {
+  await sleep(5000);
+  const frames = page.frames();
+  const targetFrame = frames.find(f => f.name() === 'iframe-old-pages');
+
+  if (!targetFrame) {
+    throw new Error('iframe: "iframe-old-pages" (used for transactions page) was not found on the page');
+  }
+
+  return targetFrame;
+}
+
 async function fetchAccounts(page: Page, startDate: Moment) {
   const accounts: TransactionsAccount[] = [];
   const accountsIds = await getAccountIdsBySelector(page);
   if (accountsIds.length <= 1) {
-    const accountData = await fetchAccountData(page, startDate);
+    const frame = await getTransactionsFrame(page);
+    const accountData = await fetchAccountData(frame, startDate);
     accounts.push(accountData);
   } else {
     for (const accountId of accountsIds) {
@@ -315,6 +328,7 @@ async function fetchAccounts(page: Page, startDate: Moment) {
   return accounts;
 }
 
+
 type ScraperSpecificCredentials = { username: string, password: string };
 
 class BeinleumiGroupBaseScraper extends BaseScraperWithBrowser<ScraperSpecificCredentials> {
diff --git a/src/scrapers/base-scraper-with-browser.ts b/src/scrapers/base-scraper-with-browser.ts
index 0a7ff5d5a..1be4a05a8 100644
--- a/src/scrapers/base-scraper-with-browser.ts
+++ b/src/scrapers/base-scraper-with-browser.ts
@@ -101,6 +101,8 @@ class BaseScraperWithBrowser<TCredentials extends ScraperCredentials> extends Ba
     this.emitProgress(ScraperProgressTypes.Initializing);
 
     const page = await this.initializePage();
+    await page.setCacheEnabled(false); // Clear cache and avoid 300's response status
+
     if (!page) {
       debug('failed to initiate a browser page, exit');
       return;
@@ -153,7 +155,7 @@ class BaseScraperWithBrowser<TCredentials extends ScraperCredentials> extends Ba
       }
 
       return browser.newPage();
-    } 
+    }
 
     const { timeout, args, executablePath, showBrowser } = this.options;
 


Reviews:
user: login: Nic3Guy
body: Fixing the lint
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-01T20:07:54Z
user: login: baruchiro
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-03-02T08:26:13Z
user: login: Nic3Guy
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-03-02T08:42:29Z

Review comments:
diff_hunk: @@ -298,11 +298,24 @@ async function getAccountIdsBySelector(page: Page): Promise<string[]> {
   return accountsIds;
 }
 
+async function getTransactionsFrame(page: Page): Promise<Frame> {
+  await sleep(5000);
+  const frames = page.frames();
+  const targetFrame = frames.find(f => f.name() === 'iframe-old-pages' || f.url().includes('some-part-of-url'));
+
+  if (!targetFrame) {
+    throw new Error('Target iframe not found');
+  }
path: src/scrapers/base-beinleumi-group.ts
user: login: baruchiro
body: Maybe more detailed and less technical message?

`iframe name='iframe-old-pages' with URL includes 'some-part-of-url' was not found on the page to login with`
created_at: 2025-03-02T08:26:12Z
author_association: COLLABORATOR
diff_hunk: @@ -298,11 +298,24 @@ async function getAccountIdsBySelector(page: Page): Promise<string[]> {
   return accountsIds;
 }
 
+async function getTransactionsFrame(page: Page): Promise<Frame> {
+  await sleep(5000);
+  const frames = page.frames();
+  const targetFrame = frames.find(f => f.name() === 'iframe-old-pages' || f.url().includes('some-part-of-url'));
+
+  if (!targetFrame) {
+    throw new Error('Target iframe not found');
+  }
path: src/scrapers/base-beinleumi-group.ts
user: login: Nic3Guy
body: Changed
created_at: 2025-03-02T08:42:29Z
author_association: CONTRIBUTOR

PR comments:
user: login: baruchiro
created_at: 2025-03-02T08:22:30Z
body: Just one question before reviewing your change, did you ran the tests with your username and password?
author_association: COLLABORATOR
user: login: Nic3Guy
created_at: 2025-03-02T08:44:30Z
body: > Just one question before reviewing your change, did you ran the tests with your username and password?

Yes, but only for "beinleumi" I didn't ran any other scraper because I didn't modified any, is that ok? 
author_association: CONTRIBUTOR
user: login: arieltobiana
created_at: 2025-03-06T07:15:00Z
body: @baruchiro—It would be great if you could insert this PR into a new release since the Benleuim scrapper has been broken for three weeks already.
Thanks!
author_association: NONE
user: login: baruchiro
created_at: 2025-03-06T07:17:06Z
body: @arieltobiana did you tested this code with your credentials too?

Sorry for the delay 😨
author_association: COLLABORATOR
user: login: arieltobiana
created_at: 2025-03-08T19:56:36Z
body: @baruchiro - I am in military reserve without the ability to test it
I suggest creating a new release with it. The scrapper is already broke, so it can't be worse :) 
author_association: NONE
user: login: github-actions[bot]
created_at: 2025-03-09T05:43:05Z
body: :tada: This PR is included in version 5.4.1 :tada:

The release is available on [GitHub release](https://github.com/eshaham/israeli-bank-scrapers/releases/tag/v5.4.1)

Your **[semantic-release](https://github.com/semantic-release/semantic-release)** bot :package::rocket:
author_association: NONE