Project: talenddataquality
Issue TDQ-11246: Amazon Redshift Data Profiling "missing system catalog data" error
When connecting to an Amazon Redshift database using the built in Postgres driver, it is not possible to perform column analysis in Data Profiling. When I click on "Select Indicators" I receive the below error after a long delay. This appears to be related to a PSQL indexing feature that is not supported in Redshift. This PSQL error seems to cause Talend to go into a loop followed by a Java Heap Error Note that this does not occur on smaller Redshift tables, but does occur 100% reproducibly on one of our larger tables. {code} !STACK 0 org.postgresql.util.PSQLException: Unable to determine a value for MaxIndexKeys due to missing system catalog data. at org.postgresql.jdbc2.AbstractJdbc2DatabaseMetaData.getMaxIndexKeys(AbstractJdbc2DatabaseMetaData.java:64) at org.postgresql.jdbc2.AbstractJdbc2DatabaseMetaData.getImportedExportedKeys(AbstractJdbc2DatabaseMetaData.java:3134) at org.postgresql.jdbc2.AbstractJdbc2DatabaseMetaData.getImportedKeys(AbstractJdbc2DatabaseMetaData.java:3464) at org.talend.metadata.managment.model.DBConnectionFillerImpl.fillPkandFk(DBConnectionFillerImpl.java:1767) at org.talend.metadata.managment.model.DBConnectionFillerImpl.fillColumns(DBConnectionFillerImpl.java:1679) at org.talend.metadata.managment.model.MetadataFillFactory.fillColumns(MetadataFillFactory.java:331) at org.talend.core.model.metadata.builder.database.DqRepositoryViewService.getColumns(DqRepositoryViewService.java:355) at org.talend.dq.nodes.DBColumnFolderRepNode.loadColumns(DBColumnFolderRepNode.java:215) at org.talend.dq.nodes.DBColumnFolderRepNode.getChildren(DBColumnFolderRepNode.java:193) at org.talend.dataprofiler.core.ui.dialog.ColumnsSelectionDialog$ModelElementContentProvider.getChildren(ColumnsSelectionDialog.java:523) at org.eclipse.ui.model.BaseWorkbenchContentProvider.getElements(BaseWorkbenchContentProvider.java:76) at org.eclipse.jface.viewers.StructuredViewer.getRawChildren(StructuredViewer.java:999) at org.eclipse.jface.viewers.ColumnViewer.getRawChildren(ColumnViewer.java:699) at org.eclipse.jface.viewers.AbstractTableViewer.getRawChildren(AbstractTableViewer.java:1043) at org.eclipse.jface.viewers.StructuredViewer.getFilteredChildren(StructuredViewer.java:905) at org.eclipse.jface.viewers.StructuredViewer.getSortedChildren(StructuredViewer.java:1057) at org.eclipse.jface.viewers.AbstractTableViewer.internalRefreshAll(AbstractTableViewer.java:667) at org.eclipse.jface.viewers.AbstractTableViewer.internalRefresh(AbstractTableViewer.java:615) at org.eclipse.jface.viewers.AbstractTableViewer.internalRefresh(AbstractTableViewer.java:607) at org.eclipse.jface.viewers.AbstractTableViewer$1.run(AbstractTableViewer.java:567) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1436) at org.eclipse.jface.viewers.StructuredViewer.preservingSelection(StructuredViewer.java:1397) at org.eclipse.jface.viewers.CheckboxTableViewer.preservingSelection(CheckboxTableViewer.java:344) at org.eclipse.jface.viewers.AbstractTableViewer.inputChanged(AbstractTableViewer.java:564) at org.eclipse.jface.viewers.ContentViewer.setInput(ContentViewer.java:292) at org.eclipse.jface.viewers.StructuredViewer.setInput(StructuredViewer.java:1675) at org.talend.dataprofiler.core.ui.dialog.TwoPartCheckSelectionDialog.setOutput(TwoPartCheckSelectionDialog.java:263) at org.talend.dataprofiler.core.ui.dialog.ColumnsSelectionDialog.selectionChanged(ColumnsSelectionDialog.java:408) at org.eclipse.jface.viewers.Viewer$2.run(Viewer.java:163) at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:42) at org.eclipse.ui.internal.JFaceUtil$1.run(JFaceUtil.java:50) at org.eclipse.jface.util.SafeRunnable.run(SafeRunnable.java:178) at org.eclipse.jface.viewers.Viewer.fireSelectionChanged(Viewer.java:160) at org.eclipse.jface.viewers.StructuredViewer.updateSelection(StructuredViewer.java:2171) at org.eclipse.jface.viewers.StructuredViewer.handleSelect(StructuredViewer.java:1202) at org.eclipse.jface.viewers.CheckboxTreeViewer.handleSelect(CheckboxTreeViewer.java:298) at org.eclipse.jface.viewers.StructuredViewer$4.widgetSelected(StructuredViewer.java:1231) at org.eclipse.jface.util.OpenStrategy.fireSelectionEvent(OpenStrategy.java:242) at org.eclipse.jface.util.OpenStrategy.access$4(OpenStrategy.java:236) at org.eclipse.jface.util.OpenStrategy$1.handleEvent(OpenStrategy.java:408) at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4353) at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1061) at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4172) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3761) at org.eclipse.jface.window.Window.runEventLoop(Window.java:832) at org.eclipse.jface.window.Window.open(Window.java:808) at org.talend.dataprofiler.core.ui.dialog.TwoPartCheckSelectionDialog.open(TwoPartCheckSelectionDialog.java:313) at org.talend.dataprofiler.core.ui.editor.analysis.ColumnAnalysisDetailsPage.openColumnsSelectionDialog(ColumnAnalysisDetailsPage.java:813) at org.talend.dataprofiler.core.ui.editor.analysis.ColumnAnalysisDetailsPage$6.mouseDown(ColumnAnalysisDetailsPage.java:474) at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:192) at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84) at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4353) at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1061) at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:4172) at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3761) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine$9.run(PartRenderingEngine.java:1151) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) at org.eclipse.e4.ui.internal.workbench.swt.PartRenderingEngine.run(PartRenderingEngine.java:1032) at org.eclipse.e4.ui.internal.workbench.E4Workbench.createAndRunUI(E4Workbench.java:148) at org.eclipse.ui.internal.Workbench$5.run(Workbench.java:636) at org.eclipse.core.databinding.observable.Realm.runWithDefault(Realm.java:332) at org.eclipse.ui.internal.Workbench.createAndRunWorkbench(Workbench.java:579) at org.eclipse.ui.PlatformUI.createAndRunWorkbench(PlatformUI.java:150) at org.talend.dataprofiler.rcp.intro.Application.start(Application.java:78) at org.eclipse.equinox.internal.app.EclipseAppHandle.run(EclipseAppHandle.java:196) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.runApplication(EclipseAppLauncher.java:134) at org.eclipse.core.runtime.internal.adaptor.EclipseAppLauncher.start(EclipseAppLauncher.java:104) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:380) at org.eclipse.core.runtime.adaptor.EclipseStarter.run(EclipseStarter.java:235) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.eclipse.equinox.launcher.Main.invokeFramework(Main.java:648) at org.eclipse.equinox.launcher.Main.basicRun(Main.java:603) at org.eclipse.equinox.launcher.Main.run(Main.java:1465) !ENTRY org.eclipse.ui 4 0 2015-11-12 11:31:39.052 !MESSAGE Unhandled event loop exception !STACK 0 java.lang.OutOfMemoryError: GC overhead limit exceeded !ENTRY org.eclipse.core.jobs 4 2 2015-11-12 11:31:39.060 !MESSAGE An internal error occurred during: "Compacting resource model". !STACK 0 java.lang.OutOfMemoryError: GC overhead limit exceeded !ENTRY org.eclipse.core.jobs 4 2 2015-11-12 11:31:39.062 !MESSAGE An internal error occurred during: "Periodic workspace save.". !STACK 0 java.lang.OutOfMemoryError: GC overhead limit exceeded {code} I have retried using the latest Redshift drivers, downloaded from here: http://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html#download-jdbc-driver With either the 4.0 or the 4.1 driver, I receive a different error: "\[Amazon\]\[JDBC\](10220) Driver not capable" I have not been able to find any information about this error on existing tickets or forums.
Story points: 3